control headers and routing depcrecated in istio 1.6 - kubernetes

I need to route my traffic based on the headers using istio but this option is deprecated in istio1.6 version. Why control headers and routing is deprecated in istio?

As mentioned in istio documentation
The mixer policy is deprecated in Istio 1.5 and not recommended for production usage.
Consider using Envoy ext_authz filter, lua filter, or write a filter using the Envoy-wasm sandbox.
Control headers and routing are not deprecated, it's just mixer which was used to do that. There are different ways to do that now, as mentioned above.
I'm not sure what exactly you want to do, but take a look at envoy filter and virtual service.
Envoy filter
There is envoy filter which add some custom headers to all the outbound responses
kind: EnvoyFilter
metadata:
name: lua-filter
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
subFilter:
name: "envoy.router"
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
"#type": "type.googleapis.com/envoy.config.filter.http.lua.v2.Lua"
inlineCode: |
function envoy_on_response(response_handle)
response_handle:logInfo(" ========= XXXXX ========== ")
response_handle:headers():add("X-User-Header", "worked")
end
And tests from curl
$ curl -s -I -X HEAD x.x.x.x/
HTTP/1.1 200 OK
server: istio-envoy
date: Mon, 06 Jul 2020 08:35:37 GMT
content-type: text/html
content-length: 13
last-modified: Thu, 02 Jul 2020 12:11:16 GMT
etag: "5efdcee4-d"
accept-ranges: bytes
x-envoy-upstream-service-time: 2
x-user-header: worked
Few links worth to check about that:
https://blog.opstree.com/2020/05/27/ip-whitelisting-using-istio-policy-on-kubernetes-microservices/
https://github.com/istio/istio/wiki/EnvoyFilter-Samples
https://istio.io/latest/docs/reference/config/networking/envoy-filter/
Virtual Service
Another thing worth to check here would be virtual service, you can do header routing based on matches here.
Take a look at example from istio documentation
HttpMatchRequest specifies a set of criterion to be met in order for the rule to be applied to the HTTP request. For example, the following restricts the rule to match only requests where the URL path starts with /ratings/v2/ and the request contains a custom end-user header with value jason.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: ratings-route
spec:
hosts:
- ratings.prod.svc.cluster.local
http:
- match:
- headers:
end-user:
exact: jason
uri:
prefix: "/ratings/v2/"
ignoreUriCase: true
route:
- destination:
host: ratings.prod.svc.cluster.local
Additionally there is my older example with header based routing in virtual service.
Let me know if you have any more questions.

Related

Override x-request-id header in istio

I was trying to understand the tracing in istio.
According to istio documentation, x-request-id can be used for tracing purpose.(https://istio.io/latest/docs/tasks/observability/distributed-tracing/overview/)
I am seeing different behavior in Istio vs pure envoy proxy.
For tracing istio and also pure envoy proxy set the x-request-id. (generated guid)
However in istio the client can send a header x-request-id and the same is forwarded to the microservices.
Whereas if I have pure envoy - the x-request-id sent by the client is not considered and envoy overrides it with a generated guid.
Can istio be configured to over-ride this x-request-id if required?
It seems it is possible to implement using envoy filter only. This is not my solution - I found it, however it looks very promising and maybe resolve your issue.
Please take a look at Istio EnvoyFilter to add x-request-id to all responses
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: gateway-response
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
subFilter:
name: "envoy.router"
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
"#type": "type.googleapis.com/envoy.config.filter.http.lua.v2.Lua"
inlineCode: |
function envoy_on_request(handle)
local metadata = handle:streamInfo():dynamicMetadata()
local headers = handle:headers()
local rid = headers:get("x-request-id")
-- for key, value in pairs(handle:headers()) do
-- handle:logTrace("key:" .. key .. " <--> value:" .. value)
-- end
if rid ~= nil then
metadata:set("envoy.filters.http.lua", "req.x-request-id", rid)
end
end
function envoy_on_response(handle)
local metadata = handle:streamInfo():dynamicMetadata():get("envoy.filters.http.lua")
local rid = metadata["req.x-request-id"]
if rid ~= nil then
handle:headers():add("x-request-id", rid)
end
end
And second one - just my assumption..
maybe you can try to add the x-request-id in the header of VirtualService? virtualservice-headers

How can I configure NGINX ingress controller to work with Cloudflare and Digital Ocean Load Balancer?

I have tried the answers in this question. This is my current configuration:
apiVersion: v1
kind: ConfigMap
metadata:
labels:
helm.sh/chart: ingress-nginx-2.13.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.35.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
data:
use-proxy-protocol: 'true'
enable-real-ip: "true"
proxy-real-ip-cidr: "173.245.48.0/20,173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/12,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32"
# use-forwarded-headers: "true"
# compute-full-forwarded-for: "true"
# forwarded-for-header: "Cf-Connecting-Ip"
# forwarded-for-header: "X-Original-Forwarded-For"
server-snippet: |
real_ip_header CF-Connecting-IP;
And none of the configuration I have tried is actually giving the originating ip as the real ip.
Before I applied the configuration, I was getting:
Host: example.com
X-Request-ID: deadcafe
X-Real-IP: 162.158.X.X (A Cloudflare IP)
X-Forwarded-For: 162.158.X.X (Same as above)
X-Forwarded-Proto: https
X-Forwarded-Host: example.com
X-Forwarded-Port: 80
X-Scheme: https
X-Original-Forwarded-For: <The Originating IP that I want>
Accept-Encoding: gzip
CF-IPCountry: IN
CF-RAY: cafedeed
CF-Visitor: {"scheme":"https"}
user-agent: Mozilla/5.0
accept-language: en-US,en;q=0.5
referer: https://pv-hr.jptec.in/
upgrade-insecure-requests: 1
cookie: __cfduid=012dadfad
CF-Request-ID: 01234faddad
CF-Connecting-IP: <The Originating IP that I want>
CDN-Loop: cloudflare
After applying the config map, the headers are:
Host: example.com
X-Request-ID: 0123fda
X-Real-IP: 10.X.X.X (An IP that matches the private ip of the Digital Ocean droplets in the vpc, so guessing its the load balancer)
X-Forwarded-For: 10.X.X.X (Same as above)
X-Forwarded-Proto: http
X-Forwarded-Host: example.com
X-Forwarded-Port: 80
X-Scheme: http
X-Original-Forwarded-For: <Originating IP>
Accept-Encoding: gzip
CF-IPCountry: US
CF-RAY: 5005deeb
CF-Visitor: {"scheme":"https"}
accept: /
user-agent: Mozilla/5.0
CF-Request-ID: 1EE7af
CF-Connecting-IP: <Originating IP>
CDN-Loop: cloudflare
So the only change after the configuration is that the real-ip now points to some internal resource on the Digital Ocean vpc. I haven't been able to track that down but I am guessing its the load balancer. I am confident that it is a DO resource because it matches the ip of the kubernetes nodes. So, I am not really sure why this is happening and what I should be doing to get the originating ip as the real ip.
The problem you are facing is here:
proxy-real-ip-cidr: "173.245.48.0/20,173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/12,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32"
However, the traffic being seen is coming from your DO LB instead 10.x.x.x. This is causing it to be ignored for this rule.
I did the following to get it functional:
apiVersion: v1
data:
enable-real-ip: "true"
server-snippet: |
real_ip_header CF-Connecting-IP;
kind: ConfigMap
metadata:
[...]
Security Notice: This will apply to all traffic even if it didn't originate from Cloudflare itself. As such, someone could spoof the headers on the request to impersonate another IP address.
I was having the same issue and the above-accepted answer didn't help me.
This what I did to solve the issue:
#1 - In ingress-nginx-controller I've added the below annotation:
# The below will instruct DO LOadBalancer to pass CloudFlare Ips to ingress instead of DO LoadBalancer IPs
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: 'true'
#2 - in Config Maps for ingress-nginx-controller I've added the below in data:
here we set as well use-proxy-protocol
adding set_real_ip_from (ClouDFlares Ips)
server-snippet could be as below or #real_ip_header X-Forwarded-For; see for more
Plese use this link to get updated cloudflare Ips list
data:
use-proxy-protocol: 'true'
server-snippet: 'real_ip_header CF-Connecting-IP;'
set_real_ip_from: '173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32'

Service Mesh: Using Istio to route TCP traffic based on Client IP in Virtual Service

Ingress gateway is located behind AWS ELB(classic) using nodeport and I want to route TCP traffic in Virtual Service based on client ip.
Of course Proxy Protocol of ELB is enabled.
When I use HTTP, it works. The configuration is below.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: app-vservice
namespace: test
spec:
hosts:
- "app-service"
http:
- match:
- headers:
x-forwarded-for:
exact: 123.123.123.123
route:
- destination:
host: app-service
subset: v2
- route:
- destination:
host: app-service
subset: v1
But I can't find headers field of TCP route in official documents.
Is it impossible?
Thank you.
According to docs yes there is no field to pass headers in TCPRoute in Istio. Also to answer your question every header manipulation should be done using envoy filters because Istio, built on envoy supports that and also decreases the complexity.
Using envoy and lua filters as stated in Istio docs. It can be achieved. Please follow envoy docs.
Checkout the Istio Discussion for headers in Virtual Service.
For implementation of the same using Lua. And a blog showing an example how to implement filters on envoy.

Istio: Can I add randomly generated unique value as a header to every request before it reaches my application

I have a RESTful service within a spring boot application. This spring boot app is deployed inside a kubernetes cluser and we have Istio as a service mesh attached to the sidecar of each container pod in the cluster. Every request to my service first hits the service mesh i.e Istio and then gets routed accordingly.
I need to put a validation for a request header and if that header is not present then randomly generate a unique value and set it as a header to the request. I know that there is Headers.HeaderOperations which i can use in the destination rule but how can i generate a unique value every time the header is missing? I dont want to write the logic inside my application as this is a general rule to apply for all the applications inside the cluster
There is important information that needs to be said in this subject. And it looks to me like You are trying to make a workaround tracing for an applications that does not forward/propagate headers in Your cluster. So I am going to mention few problems that can be encountered with this solution (just in case).
As mentioned in answer from Yuri G. You can configure unique x-request-id headers but they will not be very useful in terms of tracing if the requests are passing trough applications that do not propagate those x-request-id headers.
This is because tracing entire request paths needs to have unique x-request-id though out its entire trace. If the x-request-id value is different in various parts of the path the request takes, how are We going to put together the entire trace path?
In a scenario where two requests are received in application pod at the same time even if they had unique x-request-id headers, only application is able to tell which inbound request matches with which outbound connection. One of the requests could take longer to process and without forwarded trace header we can't tell which one is which.
Anyway for applications that do support forwarding/propagating x-request-id headers I suggest following guide from istio documentation.
Hope it helps.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: enable-envoy-xrequestid-in-response
namespace: istio-system
spec:
configPatches:
- applyTo: NETWORK_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
patch:
operation: MERGE
value:
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
always_set_request_id_in_response: true
From reading the documentation of istio and envoy it seems like this is not supported by istio/envoy out of the box. As a workaround you have 2 options
Option 1: To set the x-envoy-force-trace header in virtual service
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews-route
spec:
hosts:
- reviews.prod.svc.cluster.local
http:
- headers:
request:
set:
x-envoy-force-trace: true
It will generate a header x-request-id if it is missing. But it seems like abuse of tracing mechanism.
Option 2: To use consistentHash balancing based on header, e.g:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: bookinfo-ratings
spec:
host: ratings.prod.svc.cluster.local
trafficPolicy:
loadBalancer:
consistentHash:
httpHeaderName:
name: x-custom-request-id
It will generate the header x-custom-request-id for any request that doesn't have this header. In this case the requests with same x-custom-request-id value will go always to the same pod that can cause uneven balancing.
The answer above works well! I have updated it for the latest istio (filter name is in full):
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: enable-envoy-xrequestid-in-response
namespace: istio-system
spec:
configPatches:
- applyTo: NETWORK_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
patch:
operation: MERGE
value:
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
always_set_request_id_in_response: true

Exposing virtual service with istio and mTLS globally enabled

I've this configuration on my service mesh:
mTLS globally enabled and meshpolicy default
simple-web deployment exposed as clusterip on port 8080
http gateway for port 80 and virtualservice routing on my service
Here the gw and vs yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: http-gateway
spec:
selector:
istio: ingressgateway # Specify the ingressgateway created for us
servers:
- port:
number: 80 # Service port to watch
name: http-gateway
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: simple-web
spec:
gateways:
- http-gateway
hosts:
- '*'
http:
- match:
- uri:
prefix: /simple-web
rewrite:
uri: /
route:
- destination:
host: simple-web
port:
number: 8080
Both vs and gw are in the same namespace.
The deployment was created and exposed with these commands:
k create deployment --image=yeasy/simple-web:latest simple-web
k expose deployment simple-web --port=8080 --target-port=80 --name=simple-web
and with k get pods I receive this:
pod/simple-web-9ffc59b4b-n9f85 2/2 Running
What happens is that from outside, pointing to ingress-gateway load balancer I receive 503 HTTP error.
If I try to curl from ingressgateway pod I can reach the simple-web service.
Why I can't reach the website with mTLS enabled? What's the correct configuration?
As #suren mentioned in his answer this issue is not present in istio version 1.3.2 . So one of solutions is to use newer version.
If you chose to upgrade istio to newer version please review documentation 1.3 Upgrade Notice and Upgrade Steps as Istio is still in development and changes drastically with each version.
Also as mentioned in comments by #Manuel Castro this is most likely issue addressed in Avoid 503 errors while reconfiguring service routes and newer version simply handles them better.
Creating both the VirtualServices and DestinationRules that define the
corresponding subsets using a single kubectl call (e.g., kubectl apply
-f myVirtualServiceAndDestinationRule.yaml is not sufficient because the resources propagate (from the configuration server, i.e.,
Kubernetes API server) to the Pilot instances in an eventually
consistent manner. If the VirtualService using the subsets arrives
before the DestinationRule where the subsets are defined, the Envoy
configuration generated by Pilot would refer to non-existent upstream
pools. This results in HTTP 503 errors until all configuration objects
are available to Pilot.
It should be possible to avoid this issue by temporarily disabling mTLS or by using permissive mode during the deployment.
I just installed istio-1.3.2, and k8s 1.15.1, to reproduced your issue, and it worked without any modifications. This is what I did:
0.- create a namespace called istio and enable sidecar injection automatically.
1.- $ kubectl run nginx --image nginx -n istio
2.- $ kubectl expose deploy nginx --port 8080 --target-port 80 --name simple-web -n istio
3.- $kubectl craete -f gw.yaml -f vs.yaml
Note: these are your files.
The test:
$ curl a.b.c.d:31380/simple-web -I
HTTP/1.1 200 OK
server: istio-envoy
date: Fri, 11 Oct 2019 10:04:26 GMT
content-type: text/html
content-length: 612
last-modified: Tue, 24 Sep 2019 14:49:10 GMT
etag: "5d8a2ce6-264"
accept-ranges: bytes
x-envoy-upstream-service-time: 4
[2019-10-11T10:04:26.101Z] "HEAD /simple-web HTTP/1.1" 200 - "-" "-" 0 0 6 4 "10.132.0.36" "curl/7.52.1" "4bbc2609-a928-9f79-9ae8-d6a3e32217d7" "a.b.c.d:31380" "192.168.171.73:80" outbound|8080||simple-web.istio.svc.cluster.local - 192.168.171.86:80 10.132.0.36:37078 - -
And to be sure mTLS was enabled, this is from ingress-gateway describe command:
--controlPlaneAuthPolicy
MUTUAL_TLS
So, I don't know what is wrong, but you might want to go through these steps and discard things.
Note: the reason I am attacking istio gateway on port 31380 is because my k8s is on VMs right now, and I didn't want to spin up a GKE cluster for a test.
EDIT
Just deployed another deployment with your image, exposed it as simple-web-2, and worked again. May be I'm lucky with istio:
$ curl a.b.c.d:31380/simple-web -I
HTTP/1.1 200 OK
server: istio-envoy
date: Fri, 11 Oct 2019 10:28:45 GMT
content-type: text/html
content-length: 354
last-modified: Fri, 11 Oct 2019 10:28:46 GMT
x-envoy-upstream-service-time: 4
[2019-10-11T10:28:46.400Z] "HEAD /simple-web HTTP/1.1" 200 - "-" "-" 0 0 5 4 "10.132.0.36" "curl/7.52.1" "df0dd00a-875a-9ae6-bd48-acd8be1cc784" "a.b.c.d:31380" "192.168.171.65:80" outbound|8080||simple-web-2.istio.svc.cluster.local - 192.168.171.86:80 10.132.0.36:42980 - -
What's your k8s environment?
EDIT2
# istioctl authn tls-check curler-6885d9fd97-vzszs simple-web.istio.svc.cluster.local -n istio
HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
simple-web.istio.svc.cluster.local:8080 OK mTLS mTLS default/ default/istio-system