Client Pod identity in http request header while internal service call - kubernetes

I have deployed an application on Kubernetes and exposed with Istio service mesh. There is 2 components in the application, UI and API. I am trying to setup canary based setup to enable AB testing. So, for these 2 components, there is 2 versions (v1 and v2) has deployed, so (min) 4 pods are running.
Assume, v1 is stable and v2 is release version. Version v1 will serve real internet traffic and version v2 will serve the request from specific ip address to make sure promotion of version v2 will not impact real production environment. Refer the attached image for clarity of traffic flow within application.
Testing of UI V2 (release version) is quite easy by filter real client ip address of user using virtualService-
- headers:
x-forwarded-for:
regex: .*1.2.3.4.*
Testing of API v2 (release version) is complex, because it is not exposed to internet and must serve traffic only from UI v2 (release version) internally but I am unable to do.
url = "http://service-api"
hdr = { 'CALLER_POD' : 'ui_v2_pod_hostname_with_release' }
req = urllib.request.Request(url, headers=hdr)
response = urllib.request.urlopen(req)
One hacky trick I have applied in application, added custom http request headers "CALLER_POD" while calling API from UI v2 pod, so that API virtualService can filter out request based on "CALLER_POD". But it looks more complex because it need code refactoring on a broader level and more human manageable in future if any changes come.
Is there any way to add UI v2 pod identity (preferable hostname) in http request header while calling API service internally on Kubernetes or Istio level.

Have you tried using sourceLabels based routing? For example:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: socks-com
spec:
hosts:
- sock.com
http:
- match:
- sourceLabels:
UI: v2
- route:
- destination:
host: API
label: v2
- route:
- destination:
host: API
subset: v1
It would also require DestinationRule update with two subsets.

Related

How to replicate some traffic on Kubernetes and divert it to the Service for investigation

I'm using istio and I know that I can define weights in Virtual Service and divert traffic to different services.
My question is: how to amplify some of the traffic and direct the amplified traffic to the validation service? This amplified traffic will not go back to the original source but will be closed within the cluster. In other words, it does not bother the user.
I'm not even sure if there is an ecosystem, feature or application that provides this kind of mechanism. I don't even know if there is an ecosystem or application that provides such a mechanism and I don't know what it is called, so I'm having trouble finding it.
Thanks.
OP found a soultion by themselves, in the comments, hence the CW.
In this scenario, the best solution is to use Istio Mirroring - also called shadowing.
When traffic gets mirrored, the requests are sent to the mirrored service with their Host/Authority headers appended with -shadow. For example, cluster-1 becomes cluster-1-shadow.
Also, it is important to note that these requests are mirrored as “fire and forget”, which means that the responses are discarded.
To create mirroring rule, you have to create VirtualService with mirror and mirrorPercentage fields.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- httpbin
http:
- route:
- destination:
host: httpbin
subset: v1
weight: 100
mirror:
host: httpbin
subset: v2
mirrorPercentage:
value: 100.0
This route rule sends 100% of the traffic to v1. The last stanza specifies that you want to mirror (i.e., also send) 100% of the same traffic to the httpbin:v2 service.
[source]

Can Kubernetes Service control traffic percentage to a given pod?

Is it possible to control the percentage of traffic going to a particular pod with Kubernetes Service, without controlling the number of underlying pod? By default, kube- chooses a backend via a round-robin algorithm.
Yes, possible with the extra configuration of Service mesh.
If you looking forward to do it using the simple service it's hard to do it based on % as the default behavior is round-robin.
For example, if you are using the Istio service mesh
You can route the traffic based on weight
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
...
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v1
weight: 50
- destination:
host: reviews
subset: v3
weight: 50
where subset you can consider is more like the label so you are running the deployments with multiple labels and do the routing based on weight using the Istio.
See the example.

How to use variables in Istio VirtualService?

I'm currently working on a case when we need to dynamically create services and provide access to them via URI subpaths of the main gateway.
I'm planning to use virtual services for traffic routing for them. Virtual Service for a particular service should look like:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: subpaths-routes
spec:
hosts:
- mainservice.prod.svc.cluster.local
http:
- name: "subpath-redirection"
match:
- uri:
prefix: "/bservices/svc-2345-6789"
route:
- destination:
host: svc-2345-6789.prod.svc.cluster.local
But there may be a huge number of such services (like thousands). All follow the same pattern of routing.
I would like to know if Istio has a mechanism to specify VirtualService with variables/parameters like the following:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: subpaths-routes
spec:
hosts:
- mainservice.prod.svc.cluster.local
http:
- name: "subpath-redirection"
match:
- uri:
prefix: "/bservices/"{{ variable }}
route:
- destination:
host: {{ variable }}.prod.svc.cluster.local
In Nginx, one can do a similar thing by specifying something like this:
location ~ /service/(?<variable>[0-9a-zA-Z\_\-]+)/ {
proxy_pass http://$variable:8080;
}
Is there a way in Istio to accomplish that?
And if there is not, how would thousands of VSs impact the performance of request processing? Is It expensive to keep them in terms of CPU and RAM being consumed?
Thank you in advance!
How to use variables in Istio VirtualService?
As far as I know there is no such option in istio to specify a variable in prefix and host, if it was only a prefix then you could try with regex instead of prefix.
If you would like to automate it in some way, I mean create a variable and put in in both, prefix and host, then you could try do to it with helm.
There are few examples for virtual service in helm.
https://github.com/streamsets/helm-charts/blob/master/incubating/control-hub/templates/istio-gateway-virtualservice.yaml
https://github.com/salesforce/helm-starter-istio/blob/master/templates/virtualService.yaml
how would thousands of VSs impact the performance of request processing?
There is github issue about that, as #lanceliuu mentioned there
When we create ~1k virtualservices in a single cluster, the ingress gateway is picking up new virtualservice slowly.
So that might be one of the issues with thousands of Virtual Services.
Is It expensive to keep them in terms of CPU and RAM being consumed?
I would say it would require testing. I checked in above github issue and they mentioned that there is no mem/cpu pressure for istio components, but I can't say how expensive is that.
In theory you could create 1 big virtual service instead of thousands, but as mentioned in documentation you should rather Split large virtual services into multiple resources.
Additional resources:
https://engineering.hellofresh.com/everything-we-learned-running-istio-in-production-part-2-ff4c26844bfb
https://istio.io/latest/docs/ops/deployment/performance-and-scalability/
https://perf.dashboard.istio.io/

Is having an HTTP service as an external authorization service supported in istio?

I have seen example for EnvoyFilter in ISTIO where grpc_service is supported as filterconfig for external service call out.
kind: EnvoyFilter
metadata:
name: ext-authz
namespace: istio-system
spec:
filters:
- insertPosition:
index: FIRST
listenerMatch:
listenerType: SIDECAR_INBOUND
listenerProtocol: HTTP
filterType: HTTP
filterName: "envoy.ext_authz"
filterConfig:
grpc_service:
google_grpc:
target_uri: 127.0.0.1:9191
stat_prefix: "ext_authz"
But I want to use my external service as filterconfig using http_service instead of grpc_service but everytime I get 404 not found error.
Is http_service supported as filterConfig in Istio's envoyFilter?
version info : GKE is 14 and istio is 1.1.17
Update: modified entire answer.
After further verification it appears that Istio had http_service authorization service in the past it was not fully functional.
There were attempts to implement external HTTP service authorization for older versions of Istio, however it did work and the only workaround solutions were to use http lua filter or Nginx-Ingress Controller as Ingress Gateway that delegates the authentication part.
All of above cases can be found in this github issue. The HTTP call was successful but the headers were not being passed.
Here is another attempt in running http_service as authorization service.
As You noticed the Envoy documentation for Envoy 1.11.0 http_service filterConfig has different syntax. Therefore I suggest trying the configuration for filter from the github issue. And if It doesnt't work, try the http lua filter as a workaround.
The HTTP service as an external authorization service its not mentioned in Istio documentation so, I think its safe to say its not fully supported.

Kubernetes canary deployment routing based on geographic location

I have built a simple canary deployment in Kubernetes GCP but I dont have any control of the traffic routing , right now its random.
Ideally I would like to control the routing based on the Geographic location , however browser-agent would also do or any parameter where i can control the routing.
I followed the canary deployment technique using this link
Is there something in-built provided by k8s ?
Any pointers would be great.
Take a look at Canary Deployments using Istio.
For example, the following rule will only apply to an incoming request if it includes a “cookie” header that contains the substring “user=jason”.
apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
name: ratings-jason
spec:
destination:
name: reviews
match:
request:
headers:
cookie:
regex: "^(.*?;)?(user=jason)(;.*)?$"
...