The IP whitelisting/blacklisting example explained here https://kubernetes.io/docs/tutorials/services/source-ip/ uses source.ip attribute. However, in kubernetes (kubernetes cluster running on docker-for-desktop) source.ip returns the IP of kube-proxy. A suggested workaround is to use request.headers["X-Real-IP"], however it doesn't seem to work and returns kube-proxy IP in docker-for-desktop in mac.
https://github.com/istio/istio/issues/7328 mentions this issue and states:
With a proxy that terminates the client connection and opens a new connection to your nodes/endpoints. In such cases the source IP will always be that of the cloud LB, not that of the client.
With a packet forwarder, such that requests from the client sent to the loadbalancer VIP end up at the node with the source IP of the client, not an intermediate proxy.
Loadbalancers in the first category must use an agreed upon protocol between the loadbalancer and backend to communicate the true client IP such as the HTTP X-FORWARDED-FOR header, or the proxy protocol.
Can someone please help how can we define a protocol to get the client IP from the loadbalancer?
Maybe your are confusing with kube-proxy and istio, by default Kubernetes uses kube-proxy but you can install istio that injects a new proxy per pod to control the traffic in both directions to the services inside the pod.
With that said you can install istio on your cluster and enable it for only the services you need and apply a blacklisting using the istio mechanisms
https://istio.io/docs/tasks/policy-enforcement/denial-and-list/
To make a blacklist using the source IP we have to leave istio manage how to fetch the source IP address and use som configuration like this taken from the docs:
apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
name: whitelistip
spec:
compiledAdapter: listchecker
params:
# providerUrl: ordinarily black and white lists are maintained
# externally and fetched asynchronously using the providerUrl.
overrides: ["10.57.0.0/16"] # overrides provide a static list
blacklist: false
entryType: IP_ADDRESSES
---
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
name: sourceip
spec:
compiledTemplate: listentry
params:
value: source.ip | ip("0.0.0.0")
---
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
name: checkip
spec:
match: source.labels["istio"] == "ingressgateway"
actions:
- handler: whitelistip
instances: [ sourceip ]
---
You can use the param providerURL to maintain an external list.
Also check to use externalTrafficPolicy: Local on the ingress-gateway servce of istio.
As per comments my last advice is to use a different ingress-controller to avoid the use of kube-proxy, my recomendation is to use the nginx-controller
https://github.com/kubernetes/ingress-nginx
You can configure this ingress as a regular nginx acting as a proxy
Related
I have this service that limits IPs to 2 requests per day running in Kubernetes.
Since it is behind an ingress proxy the request IP is always the same, so it is limiting he total amount of requests to 2.
Its possible to turn on proxy protocol with a config like this:
apiVersion: v1
metadata:
name: nginx-ingress-controller
data:
use-proxy-protocol: "true"
kind: ConfigMap
But this would turn it on for all services, and since they don't expect proxy-protocol they would break.
Is there a way to enable it for only one service?
It is possible to configure Ingress so that it includes the original IPs into the http header.
For this I had to change the service config.
Its called ingress-nginx-ingress-controller(or similar) and can be found with kubectl get services -A
spec:
externalTrafficPolicy: Local
And then configure the ConfigMap with the same name:
data:
compute-full-forwarded-for: "true"
use-forwarded-headers: "true"
Restart the pods and then the http request will contain the fields X-Forwarded-For and X-Real-Ip.
This method won't break deployments not expecting proxy-protocol.
I am watching a Pluralsight video on the Istio service mesh. One part of the presentation says this:
The VirtualService uses the Kubernetes service to find the IP addresses of all the pods. The VirtualService doesn't route any traffic through the [Kubernetes] service, but it just uses it to get the list of endpoints where the traffic could go.
And it shows this graphic (to show the pod discovery, not for traffic routing):
I am a bit confused by this because I don't know how an Istio VirtualService knows which Kubernetes Service to look at. I don't see any reference in the example Istio VirtualService yaml files to a Kubernetes Service.
I have theorized that the DestinationRules could have enough labels on them to get down to just the needed pods, but the examples only use the labels v1 and v2. It seems unlikely that a version alone will give only the needed pods. (Many different Services could be on v1 or v2.)
How does an Istio VirtualService know which Kubernetes Service to associate to?
or said another way,
How does an Istio VirtualService know how to find the correct pods from all the pods in the cluster?
When creating a VitualService you define which service to find in route.destination section
port : service running on port
host : name of the service
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: test
spec:
hosts:
- "example.com"
gateways:
- test-gateway
http:
- match:
- uri:
prefix: /
route:
- destination:
port:
number: 80
host: app-service
so,
app-pod/s -> (managed by) app-service -> test virtual service
Arfat's answer is correct.
I want to add the following part from the docs about the host, which should make things even more clear.
https://istio.io/latest/docs/reference/config/networking/virtual-service/#VirtualService
[...] Note for Kubernetes users: When short names are used (e.g. “reviews” instead of “reviews.default.svc.cluster.local”), Istio will interpret the short name based on the namespace of the rule, not the service. A rule in the “default” namespace containing a host “reviews” will be interpreted as “reviews.default.svc.cluster.local”, irrespective of the actual namespace associated with the reviews service. To avoid potential misconfigurations, it is recommended to always use fully qualified domain names over short names.
So when you write host: app-service and the VirtualService is in the default namespace, the host is interpreted as app-service.default.svc.cluster.local, which is the FQDN of the kubernetes service. If the app-service is in another namespace, say dev, you need to set the host as host: app-service.dev.svc.cluster.local.
Same goes for DestinationRule, where the FQDN of a kubernetes service is defined as host, as well.
https://istio.io/latest/docs/reference/config/networking/destination-rule/#DestinationRule
VirtualService and DestinationRule are configured for a host. The VirtualService defines where the traffic should go (eg host, weights for different versions, ...) and the DestinationRule defines, how the traffic should be handled, (eg load balancing algorithm and how are the versions defined.
So traffic is not routed like this
Gateway -> VirtualService -> DestinationRule -> Service -> Pod, but like
Gateway -> Service, considering the config from VirtualService and DestinationRule.
I've gotten up Traefik as an Ingress in Kubernetes with this configuration: https://github.com/RedxLus/traefik-simple-kubernetes/tree/master/V1.7
And works well to HTTP and HTTPS but I don't know how can open others ports to forward, for example, a Pod with an Ingress with MySQL in port 3306
Thanks for every answer!
Traefik doesn't support it if you are using an Ingress resource and that resource doesn't support L4 type of traffic like mentioned in the other answer.
But if you are using an Nginx ingress controller there is a workaround, use a ConfigMap with the ingress controller options --tcp-services-configmap and --udp-services-configmap as described here. Then your tcp-services ConfigMap would look something like this:
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-services
namespace: ingress-nginx
data:
9000: "default/example-go:8080"
The advantage of this is having a single entry point to your cluster (this applies to any ingress that would be used for TCP/UDP) but the downside is overhead of having an extra layer compared to just simply having a Kubernetes Service (NodePort or LoadBalancer) that already listens on TCP/UDP ports.
Kubernetes Ingress API does not support it. But it is possible to use Traefik as TCP proxy for your desired use-case, but only, if you make use of TLS encrypted connections. Otherwise, based on the level 4 protocol, it's not possible to distinguish between the different hostnames and you would have to use one entrypoint per TCP router. Check this issue in GitHub.
I tried whitelisting IP address/es in my kubernetes cluster's incoming traffic using this example :
Although this works as expected, wanted to go a step further and try if I can use istio gateways or virtual service when I set up Istio rule, instead of Loadbalancer(ingressgateway).
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
name: checkip
namespace: my-namespace
spec:
match: source.labels["app"] == "my-app"
actions:
- handler: whitelistip.listchecker
instances:
- sourceip.listentry
---
Where my-app is of kind: Gateway with a certain host and port, and labelled app=my-app.
Am using istio version 1.1.1
Also my cluster has all the istio-system running with envoy sidecars on almost all service pods.
You confuse one thing that, in above rule, match: source.labels["app"] == "my-app" is not referring to any resource's label, but to pod's label.
From OutputTemplate Documentation:
sourceLabels | Refers to source pod labels.
attributebindings can refer to this field using $out.sourcelabels
And you can verify by looking for resources with "app=istio-ingressgateway" label via:
kubectl get pods,svc -n istio-system -l "app=istio-ingressgateway" --show-labels
You can check this blog from istio about Mixer Adapter Model, to understand complete mixer model, its handlers,instances and rules.
Hope it helps!
My goal is to make my web application (deployed on Kubernetes 1.4 cluster) see the IP of the client that originally made the HTTP request. As I'm planning to run the application on a bare-metal cluster, GCE and the service.alpha.kubernetes.io/external-traffic: OnlyLocal service annotation introduced in 1.4 is not applicable for me.
Looking for alternatives, I've found this question which is proposing to set up an Ingress to achieve my goal. So, I've set up the Ingress and the NginX Ingress Controller. The deployment went smoothly and I was able to connect to my web app via the Ingress Address and port 80. However in the logs I still see cluster-internal IP (from 172.16.0.0/16) range - and that means that the external client IPs are not being properly passed via the Ingress. Could you please tell me what do I need to configure in addition to the above to make it work?
My Ingress' config:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myWebApp
spec:
backend:
serviceName: myWebApp
servicePort: 8080
As a layer 4 proxy, Nginx cannot retain the original source IP address in the actual IP packets. You can work around this using the Proxy protocol (the link points to the HAProxy documentation, but Nginx also supports it).
For this to work however, the upstream server (meaning the myWebApp service in your case) also needs to support this protocol. In case your upstream application also uses Nginx, you can enable proxy protocol support in your server configuration as documented in the official documentation.
According to the Nginx Ingress Controller's documentation, this feature can be enabled in the Ingress Controller using a Kubernetes ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-ingress-controller
data:
use-proxy-protocol: "true"
Specify the name of the ConfigMap in your Ingress controller manifest, by adding the --nginx-configmap=<insert-configmap-name> flag to the command-line arguments.