Is there a way to prevent envoy from adding specific headers? - kubernetes

Question

According to the docs here https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-proto
Envoy proxy adds the Header X-Forwarded-Proto to the request, for some reason the header value is wrong; it set it as http although the incoming requests scheme is https which cause some problems in my application code since it depends on the correct value of this header.
Is this a bug in envoy? Can I prevent envoy from doing this?

Answer 1

As I mentioned in comments there is related github issue about that.
Is there a way to prevent envoy from adding specific headers?
There is istio dev #howardjohn comment about that
We currently have two options:
EnvoyFilter
Alpha api
There will not be a third; instead we will promote the alpha API.
So the first option would be envoy filter.
There are 2 answers with that in above github issue.
Answer provided by #jh-sz
In general, use_remote_address should be set to true when Envoy is deployed as an edge node (aka a front proxy), whereas it may need to be set to false when Envoy is used as an internal service node in a mesh deployment.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: xff-trust-hops
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: NETWORK_FILTER
match:
context: ANY
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
patch:
operation: MERGE
value:
typed_config:
"#type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"
use_remote_address: true
xff_num_trusted_hops: 1
AND
Answer provided by #vadimi
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: my-app-filter
spec:
workloadLabels:
app: my-app
filters:
- listenerMatch:
portNumber: 5120
listenerType: SIDECAR_INBOUND
filterName: envoy.lua
filterType: HTTP
filterConfig:
inlineCode: |
function envoy_on_request(request_handle)
request_handle:headers():replace("x-forwarded-proto", "https")
end
function envoy_on_response(response_handle)
end
The second option would be Alpha api, this feature is actively in development and is considered pre-alpha.
Istio provides the ability to manage settings like X-Forwarded-For (XFF) and X-Forwarded-Client-Cert (XFCC), which are dependent on how the gateway workloads are deployed. This is currently an in-development feature. For more information on X-Forwarded-For, see the IETF’s RFC.
You might choose to deploy Istio ingress gateways in various network topologies (e.g. behind Cloud Load Balancers, a self-managed Load Balancer or directly expose the Istio ingress gateway to the Internet). As such, these topologies require different ingress gateway configurations for transporting correct client attributes like IP addresses and certificates to the workloads running in the cluster.
Configuration of XFF and XFCC headers is managed via MeshConfig during Istio installation or by adding a pod annotation. Note that the Meshconfig configuration is a global setting for all gateway workloads, while pod annotations override the global setting on a per-workload basis.

Answer 2

The reason this happens is most likely because you have one or more proxies in front of Envoy/Istio.
You need to tell Envoy how many proxies you have in front of it so that it can set forwarded headers correctly (such as X-Forwarded-Proto and X-Forwarded-For).
In Istio 1.4+ you can achieve this with an Envoy filter:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: xff-trust-hops
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: NETWORK_FILTER
match:
context: ANY
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
patch:
operation: MERGE
value:
typed_config:
"#type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"
use_remote_address: true
xff_num_trusted_hops: 1 # Change as needed
Note that if you have multiple proxies in front of Envoy you have to change the xff_num_trusted_hops variable to the correct amount. For example if you have a GCP or AWS cloud load balancer, you might have to increase this value to 2.
In Istio 1.8+, you will be able to configure this via the Istio operator instead, example:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
defaultConfig:
gatewayTopology:
numTrustedProxies: 1 # Change as needed
More information is available here.