Kubernetes Traefik Ingress using WebSockets sending 500's - kubernetes

Hello and thanks for reading!
We've been trying to setup a secure web socket via traefic to a spring webserver.
The connection is inbound to traefik via HTTP/2. The IngressRoute (HttpRouter) responsible for this endpoint is doing the SSL termination and moving the traffic to a TraefikService with scheme http.
We tried several variants, with middleware which append Headers like "X-Forwarded-Proto: https", but still the furthest we could get is as follows: The Spring WebServer did accept the connection, but it instantly got terminated with SC 500 again and no response was sent to the client.
Am I missing any configuration for the setup as described above?
Note: Traefik does not show any error logs and the Spring webservice only tells that the connection was interrupted.
Configuration snippets (commented out parts were tried before but didn't work either)
Middleware:
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: wss-headers
namespace: rc
spec:
headers:
customRequestHeaders:
Connection: "keep-alive, Upgrade"
# Connection: "Upgrade"
# X-Forwarded-Proto: "https"
Upgrade: "WebSocket"
# X-Forwarded-Port: "443"
WSS ingress:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: uiserver-ws-ing
namespace: rc
spec:
tls:
secretName: tls-example.org
options:
namespace: rc
domains:
- main: example.org
sans:
- example.com
routes:
- kind: Rule
priority: 100
match: Host(`example.org`) && PathPrefix(`/ws`)
# middlewares:
# - name: wss-headers
services:
- name: uiserver-svc
namespace: rc
port: 8080
scheme: http
# scheme: h2c
Thank you & Best Regards,
Alex

Related

Istio break SFTP communication

I have a simple stateful with SFTP features and a second service try to connect to it.
When ISTIO is enabled, the connection is closed by the sftp service.
We can find this log:
Bad protocol version identification '\026\003\001'
The routing is OK.
The service:
apiVersion: v1
kind: Service
metadata:
name: foo
namespace: bar
spec:
ports:
- port: 22
targetPort: 22
protocol: TCP
name: tcp-sftp
selector:
app.kubernetes.io/instance: foo-bar
I try to add a VirtualService with no luck:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: foo-bar-destination-virtualservice
namespace: bar
spec:
hosts:
- foo.bar.svc.cluster.local
tcp:
- match:
- port: 22
route:
- destination:
host: foo.bar.svc.cluster.local
port:
number: 22
The workaround is to disabled the sidecar on the sftp pod for now:
sidecar.istio.io/inject: "false"
It seems Envoy proxy (and Istio by proxy) does not support SFTP protocol (referecce).
Your workaround is currently the only way to make it work.
If you want your auto-discovered services in the mesh to route/access your SFTP service, you can additionally create ServiceEntry pointing to it.

Istio 1.5 cors not working - Response to preflight request doesn't pass access control check

Cors preflight requests do not work when a Jwt Policy is configured on the istio-ingressgateway target.
Gateway
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: api-gateway
namespace: foo
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "api.example.com"
tls:
httpsRedirect: true # sends 301 redirects for http requests
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
privateKey: /etc/istio/ingressgateway-certs/tls.key
hosts:
- "api.example.com"
Virtual Service
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: backend-vs
namespace: foo
spec:
hosts:
- "api.example.com"
gateways:
- api-gateway
http:
- match:
- uri:
prefix: /api/v1/info
route:
- destination:
host: backend.foo.svc.cluster.local
corsPolicy:
allowOrigin:
- "https://app.example.com"
allowMethods:
- POST
- GET
- PUT
- DELETE
- PATCH
- OPTIONS
allowHeaders:
- authorization
- content-type
- accept
- origin
- user-agent
allowCredentials: true
maxAge: 300s
Security
apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
name: "jwt-example"
namespace: foo
spec:
selector:
matchLabels:
app: backend
jwtRules:
- issuer: "http://keycloak.foo/auth/realms/example"
jwksUri: "http://keycloak.foo/auth/realms/example/protocol/openid-connect/certs"
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: require-jwt-example
namespace: foo
spec:
selector:
matchLabels:
app: backend
action: ALLOW
rules:
- from:
- source:
requestPrincipals: ["http://keycloak.foo/auth/realms/example/http://keycloak.foo/auth/realms/example"]
when:
- key: request.auth.claims[groups]
values: ["group1"]
when I test the web application in firefox it works fine, but in other browsers like opera, chrome, safari, it fails with the following error:
Access to XMLHttpRequest at 'https://api.example.com/api/v1/info' from origin 'https://app.example.com' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
What makes me more thoughtful is because in firefox it works well but in other browsers it fails
NOTE: To validate that the cors policy was correct in istio, what I did was disable this policy in istio and test in firefox to see what was happening, the result was that a problem with cors did indeed come out, but when I re-enabled the cors in istio when rerunning in firefox the request works fine.
Good after doing segmented tests and see what was causing the error, I found that the problem appeared when I created the keycloak gateway (keycloak.example.com) that was running on the same service port (backend.example.com) , which by default for https is 443 and for http is 80.
What I did was expose keycloak to another port on the gateway (ingressgateway). with the above and the angular application I stop putting problem of the cors.

Istio Ingress Gateway with TLS termination returning 503 service unavailable

We want to to route https traffic to an https endpoint using Istio Ingress Gateway.
We terminate the TLS traffic at the Ingress Gateway, but our backend service uses https as well.
I have the following manifests:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: datalake-dsodis-istio-gateway
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- "gw-hdfs-spark.dsodis.domain"
- "spark-history.dsodis.domain"
port:
name: https-wildcard
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
privateKey: /etc/istio/ingressgateway-certs/tls.key
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: gw-spark-history-istio-vs
spec:
gateways:
- default/datalake-dsodis-istio-gateway
hosts:
- "spark-history.dsodis.domain"
http:
- match:
- uri:
prefix: /
route:
- destination:
host: gateway-svc-clusterip.our_application_namespace.svc.cluster.local
port:
number: 8443
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: originate-tls-for-spark-history
spec:
host: gateway-svc-clusterip.our_application_namespace.svc.cluster.local
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 8443
tls:
mode: SIMPLE
The problem is most likely, that we are sending TLS terminated traffic, (so to say) HTTP traffic, to the HTTPS backend. Therefore we might get 503 Service Unavailable when accessing the service through Istio.
The command accessing it is:
curl -vvvv -H"Host: spark-history.dsodis.domain" --resolve "spark-history.dsodis.domain:31390:IP" https://spark-history.dsodis.domain:31390/gateway/default/sparkhistory -k
My question is, how can I tell Istio to route traffic to the backend service using https?
Thanks in advance.
Best regards,
rforberger
As RonnyForberger mentioned in his comment this can be achieved by creating DestinationRule that tells the traffic to the destination service to be TLS connection.
So in this scenario:
HTTPS request gets TLS terminated at GateWay to HTTP.
Then the HTTP request is translated to TLS with DestinationRule to HTTPS.
HTTPS request reaches HTTPS backend.

Cannot allow external traffic through ISTIO

I am trying to setup Istio and I need to whitelist few ports for allowing non mTLS traffic from outside world coming in through specfic port for few pods runnings in local k8s.
I am unable to find a successful way of doing it.
Tried Service entry, policy and destination rule and didnt succeed.
Helps is highly appreciated.
version.BuildInfo{Version:"1.1.2", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"35adf5bb-5570-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.1"}```
Service Entry
```apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: external-traffic
namespace: cloud-infra
spec:
hosts:
- "*.cluster.local"
ports:
- number: 50506
name: grpc-xxx
protocol: TCP
location: MESH_EXTERNAL
resolution: NONE```
You need to add a DestinationRule and a Policy :
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: destinationrule-test
spec:
host: service-name
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
portLevelSettings:
- port:
number: 8080
tls:
mode: DISABLE
---
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: policy-test
spec:
targets:
- name: service-name
ports:
- number: 8080
peers:
This has been tested with istio 1.0, but it will probably work for istio 1.1. It is heavily inspired by the documentation https://istio.io/help/ops/setup/app-health-check/
From your question, I understood that you want to control your ingress traffic allow some ports to your services that functioning in your mesh/cluster from outside, but your configuration is for egress traffic.
In order to control and allow ports to your services from outside, you can follow these steps.
1.Make sure that containerPort included to your deployment/pod configuration.
For more info
2.You have to have service pointing to your backends/pods. For more info about Kubernetes Services.
3.Then in your Istio enabled cluster, you have to create Gateway similar to below configuration:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: your-service-gateway
namespace: foo-namespace # Use same namespace with backend service
spec:
selector:
istio: ingressgateway # use Istio default gateway implementation
servers:
- port:
number: 80
name: HTTP
protocol: HTTP
hosts:
- "*"
4.Then configure route to your service for traffic entering via the this gateway by creating VirtualService:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: your-service
namespace: foo-namespace # Use same namespace with backend service
spec:
hosts:
- "*"
gateways:
- your-service-gateway # define gateway name
http:
- match:
- uri:
prefix: "/"
route:
- destination:
port:
number: 3000 # Backend service port
host: your-service # Backend service name
Hope it helps.

Mounting applications of different paths with Istio

We have installed Istio on my Kubernetes cluster on AWS (we are using EKS). We have deployed several applications such as: Airflow, Jenkins, Grafana, etc. and we are able to reach them with port-forward. So, they are working as expected.
Now, what we would like to achieve is the possibility of mounting the applications on specific paths so that we can access them via an unique entrypoint.
Here an example to explain what I mean with "unique entrypoint":
http://a517f7.eu-west-1.elb.amazonaws.com/aiflow/ and you can see Airflow dahsboard
http://a517f7.eu-west-1.elb.amazonaws.com/jenkins/ and you can see Jenkins dahsboard
http://a517f7.eu-west-1.elb.amazonaws.com/grafana/ and you can see Grafana dahsboard
What we tried is the following
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: apps-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: airflow-virtual-service
spec:
hosts:
- "*"
gateways:
- apps-gateway
http:
- match:
- uri:
prefix: /airflow
route:
- destination:
host: webserver.airflow.svc.cluster.local
port:
number: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grafana-virtual-service
spec:
hosts:
- "*"
gateways:
- apps-gateway
http:
- match:
- uri:
prefix: /grafana
route:
- destination:
host: grafana.grafana.svc.cluster.local
port:
number: 3306
---
and so on
In this way but I keep having Aiflow 404 = lots of circles or similar depending on the service.
Do you know how to achieve such a result with Istio? We are also open to also use Nginx or Traefik.
So based on the comments:
The best way to expose a frontend & backend it's to separate both with a wildcard domain like *.domain.com.
For the frontend:
my-app.domain.com/
For the backend services it will always start with a subdomain like api:
api.domain.com/foo
For backend service you will need to use rewrite if your backend service it's no serving the path.
rewrite:
uri: "/"
Check Istio docs for more info about rewrite https://istio.io/docs/reference/config/istio.networking.v1alpha3/#Destination