I have problem related to WebSocket connection on - Istio Ingress Gateway
My cluster:
Istio - 1.7.2, Kubernetes - 1.18.6
I’m trying to run my application on new config cluster, My app is working properly on Istio 1.5.1 and k8s 1.15.11.
I have one problem with properly WebSocket connection on internal IngressGateway, rest of features is working.
Config:
I using two Istio IngressGateways:
default external IngressGateway
internal IngressGateway (ClusterIP)
Example traffic flow:
Internet > external IngressGateway > Ocelot Gateway > internal IngressGateway > services (pods)
This communication is working properly for http - but websocket doesn’t working.
I found that connection is break on internal IngressGateway.
I see only this log - internal ingress gateway pod:
[2020-10-12T10:05:16.903Z] "- - HTTP/2" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.244.2.62:8080 10.244.1.16:58426 - -
I don’t see any traffic in my destination service.
I have checked websocket connection directly on my destination service and there is working properly.
I have tried change VirtualService config - add „websocketUpgrade: True” - but there isn’t this field on new Istio.
Istio installation:
I have used istioctl, my IstioOperator below:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
accessLogFile: /dev/stdout
components:
ingressGateways:
- namespace: istio-system
name: internal-ingressgateway
label:
app: internal-ingressgateway
istio: internal-ingressgateway
enabled: true
k8s:
resources:
requests:
cpu: 100m
memory: 1000Mi
service:
type: ClusterIP
ports:
- port: 80
targetPort: 8080
values:
gateways:
istio-ingressgateway:
debug: Trace
global:
istiod:
enableAnalysis: true
pilot:
env:
PILOT_ENABLE_STATUS: true
Do you have any ideas on these issues?
UPDATE - add configs:
External IngressGateway
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: external-gtw
namespace: istio-system
labels:
app: ingressgateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
protocol: HTTP2
name: http
hosts:
- "*"
tls:
httpsRedirect: true # sends 301 redirect for http requests
- port:
number: 443
protocol: HTTPS
name: https-default
tls:
mode: SIMPLE
serverCertificate: "sds"
privateKey: "sds"
credentialName: ........
hosts:
- "*"
Internal IngressGateway
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: internal-gtw
namespace: istio-system
labels:
app: internal-gtw
spec:
selector:
app: internal-ingressgateway
servers:
- port:
number: 80
protocol: HTTP2
name: http
hosts:
- "*"
VirtualService
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: notificationservice
spec:
gateways:
- istio-system/internal-gtw
hosts:
- '*'
http:
- match:
- uri:
prefix: /notificationservice/
rewrite:
uri: /
route:
- destination:
host: notificationservice
DestinationRules - currently I don't use
Related
I used below command to bring up the pod:
kubectl create deployment grafana --image=docker.io/grafana/grafana:5.4.3 -n monitoring
Then I used below command to create custerIp:
kubectl expose deployment grafana --type=ClusterIP --port=80 --target-port=3000 --protocol=TCP -n monitoring
Then I have used below virtual service:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grafana
spec:
hosts:
- "*"
gateways:
- cogtiler-gateway.skydeck
http:
- match:
- uri:
prefix: /grafana
route:
- destination:
port:
number: 3000
host: grafana
kubectl apply -f grafana-virtualservice.yaml -n monitoring
Output:
virtualservice.networking.istio.io/grafana created
Now, when I try to access it, I get below error from grafana:
**If you're seeing this Grafana has failed to load its application files
1. This could be caused by your reverse proxy settings.
2. If you host grafana under subpath make sure your grafana.ini root_path setting includes subpath
3. If you have a local dev build make sure you build frontend using: npm run dev, npm run watch, or npm run build
4. Sometimes restarting grafana-server can help **
The easiest and working out of the box solution to configure that would be with a grafana host and / prefix.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: grafana-gateway
namespace: monitoring
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http-grafana
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grafana-vs
namespace: monitoring
spec:
hosts:
- "grafana.example.com"
gateways:
- grafana-gateway
http:
- match:
- uri:
prefix: /
route:
- destination:
host: grafana
port:
number: 80
As you mentioned in the comments, I want to use path based routing something like my.com/grafana, that's also possible to configure. You can use istio rewrite to configure that.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: grafana-gateway
namespace: monitoring
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http-grafana
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grafana-vs
namespace: monitoring
spec:
hosts:
- "*"
gateways:
- grafana-gateway
http:
- match:
- uri:
prefix: /grafana
rewrite:
uri: /
route:
- destination:
host: grafana
port:
number: 80
But, according to this github issue you would have also additionally configure grafana for that. As without the proper grafana configuration that won't work correctly.
I found a way to configure grafana with different url with the following env variable GF_SERVER_ROOT_URL in grafana deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: grafana
name: grafana
spec:
replicas: 1
selector:
matchLabels:
app: grafana
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: grafana
spec:
containers:
- image: docker.io/grafana/grafana:5.4.3
name: grafana
env:
- name: GF_SERVER_ROOT_URL
value: "%(protocol)s://%(domain)s/grafana/"
resources: {}
Also there is a Virtual Service and Gateway for that deployment.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: grafana-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http-grafana
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grafana-vs
spec:
hosts:
- "*"
gateways:
- grafana-gateway
http:
- match:
- uri:
prefix: /grafana/
rewrite:
uri: /
route:
- destination:
host: grafana
port:
number: 80
You need to create a Gateway to allow routing between the istio-ingressgateway and your VirtualService.
Something in the lines of :
kind: Gateway
metadata:
name: ingress
namespace: istio-system
spec:
selector:
# Make sure that the istio-ingressgateway pods have this label
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- my.domain.com
You also need a DNS entry for your domain (my-domain.com) that points to the IP address of your istio-ingressgateway.
When your browser will hit my.domain.com, then it'll be redirected to the istio-ingressgateway. The istio-ingressgateway will inspect the Host field from the request, and redirect the request to grafana (according to VirtualService rules).
You can check kubectl get svc -n istio-system | grep istio-ingressgateway to get the public IP of your ingress gateway.
If you want to enable TLS, then you need to provision a TLS certificate for your domain (most easy with cert-manager). Then you can use https redirect in your gateway, like so :
kind: Gateway
metadata:
name: ingress
namespace: whatever
spec:
selector:
# Make sure that the istio-ingressgateway pods have this label
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- my.domain.com
tls:
httpsRedirect: true
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- my.domain.com
tls:
mode: SIMPLE
# name of the secret containing the TLS certificate + keys. The secret must exist in the same namespace as the istio-ingressgateway (probably istio-system namespace)
# This secret can be created by cert-manager
# Or you can create a self-signed certificate
# and add it to manually inside the browser trusted certificates
credentialName: my-domain-tls
Then you VirtualService
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grafana
spec:
hosts:
- "my.domain.com"
gateways:
- ingress
http:
- match:
- uri:
prefix: /grafana
route:
- destination:
port:
number: 3000
host: grafana
So we have the following use case running on Istio 1.8.2/Kubernetes 1.18:
Our cluster is exposed via a External Loadbalancer on Azure. When we expose the app the following way, it works:
---
apiVersion: apps/v1
kind: ReplicaSet
metadata:
annotations:
...
name: frontend
namespace: frontend
spec:
replicas: 1
selector:
matchLabels:
app: applicationname
template:
metadata:
labels:
app: appname
name: frontend
customer: customername
spec:
imagePullSecrets:
- name: yadayada
containers:
- name: frontend
image: yadayada
imagePullPolicy: Always
ports:
- name: https
protocol: TCP
containerPort: 80
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
---
apiVersion: v1
kind: Service
metadata:
name: frontend-svc
namespace: frontend
labels:
name: frontend-svc
customer: customername
spec:
type: LoadBalancer
ports:
- name: http
port: 80
targetPort: 80
selector:
name: frontend
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: frontend
namespace: frontend
annotations:
kubernetes.io/ingress.class: istio
kubernetes.io/tls-acme: "true"
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
rules:
- host: "customer.domain.com"
http:
paths:
- path: /
pathType: Prefix
backend:
serviceName: frontend-svc
servicePort: 80
tls:
- hosts:
- "customer.domain.com"
secretName: certificate
When we start using a Virtualservice and Gateway, we fail to make it work for some reason. We wanna use VSVC and Gateways cause they offer more flexibility and options (like url rewriting). Other apps dont have this issue running on istio (much simpler as well), we dont have networkpolicy in place (yet). We simply cannot reach the webpage. Anyone has an idea? Virtualservice and Gateway down below. with the other 2 replicasets not mentioned cause they are not the problem:
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
creationTimestamp: null
name: virtualservice-name
namespace: frontend
spec:
gateways:
- frontend
hosts:
- customer.domain.com
http:
- match:
- uri:
prefix: /
route:
- destination:
host: frontend
port:
number: 80
weight: 100
- match:
- uri:
prefix: /api/
route:
- destination:
host: backend
port:
number: 8080
weight: 100
- match:
- uri:
prefix: /auth/
route:
- destination:
host: keycloak
port:
number: 8080
weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: frontend
namespace: frontend
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http2
protocol: HTTP
tls:
httpsRedirect: True
hosts:
- "customer.domain.com"
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: PASSTHROUGH
credentialName: customer-cert
hosts:
- "customer.domain.com"
Your Gateway specifies PASSTHROUGH, however your VirtualService provides an HttpRoute. This means the TLS connection is not terminated by the Gateway, but the VirtualService expects terminated TLS. See also this somewhat similar question.
How do I properly HTTPS secure an application when using Istio?
#user140547 Correct, we changed that now. But we still couldn't access the application.
We found out that one of the important services was not receiving gateway traffic, since that one wasn't setup correctly. It is our first time having an istio deployment with multiple services. So we thought each of them needed their own Gateway. Little did we know that 1 gateway was more then enough...
I'm new in the k8s world.
Im my dev enviroment, I use ngnix as proxy(with CORS configs and with headers forwarding like ) for the different microservices (all made with spring boot) I have. In a k8s cluster, I had to replace it with Istio?
I'm trying to run a simple microservice(for now) and use Istio for routing to it. I've installed istio with google cloud.
If I navigate to IstioIP/auth/api/v1 it returns 404
This is my yaml file
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: gateway
spec:
selector:
istio: ingressgateway # use Istio default gateway implementation
servers:
- port:
name: http
number: 80
protocol: HTTP
hosts:
- '*'
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: virtual-service
spec:
hosts:
- "*"
gateways:
- gateway
http:
- match:
- uri:
prefix: /auth
route:
- destination:
host: auth-srv
port:
number: 8082
---
apiVersion: v1
kind: Service
metadata:
name: auth-srv
labels:
app: auth-srv
spec:
ports:
- name: http
port: 8082
selector:
app: auth-srv
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: auth-srv
spec:
replicas: 1
template:
metadata:
labels:
app: auth-srv
version: v1
spec:
containers:
- name: auth-srv
image: gcr.io/{{MY_PROJECT_ID}}/auth-srv:1.5
imagePullPolicy: IfNotPresent
env:
- name: JAVA_OPTS
value: '-DZIPKIN_SERVER=http://zipkin:9411'
ports:
- containerPort: 8082
livenessProbe:
httpGet:
path: /api/v1
port: 8082
initialDelaySeconds: 60
periodSeconds: 5
Looks like istio doesn't know anything about the url. Therefore you are getting a 404 error response.
If you look closer at the configuration in the virtual server you have configured istio to match on path /auth.
So if you try to request ISTIOIP/auth you will reach your microservice application. Here is image to describe the traffic flow and why you are getting a 404 response.
I'm using Istio 1.1.7 and tried the following Gateway and VirtualService to the Kubernetes Dashboard.
But I'm seeing the following:
kubectl -n kube-system logs -f kubernetes-dashboard-7cf9657c74-5fsmq
2019/05/31 15:53:04 http: TLS handshake error from 10.244.3.11:54392: tls: first record does not look like a TLS handshake
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: default-ingressgateway
namespace: default
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- "*"
port:
name: http
number: 80
protocol: HTTP
- hosts:
- "*"
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
privateKey: /etc/istio/ingressgateway-certs/tls.key
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: dashboard-virtualservice
namespace: kube-system
spec:
hosts:
- "*"
gateways:
- default/default-ingressgateway
http:
- match:
- uri:
prefix: "/dashboard"
rewrite:
uri: "/"
route:
- destination:
host: kubernetes-dashboard
port:
number: 443
NOTE: I've also tried change the Gateway Server-TLSOptions-TLSmode from mode: SIMPLE to mode mode: PASSTHROUGH, but PASSTHROUGH I just get We can’t connect to the server in my Browser.
I think PASSTHROUGH should work, if you configure it properly. Here is an example that does something similar. https://istio.io/docs/examples/advanced-gateways/ingress-sni-passthrough/
I am using helm to install istio-1.0.0 version with --set grafana.enabled=true.
To access the grafana dashboard, I have to do port forwarding using kubectl command. It works okay. However, i want to access it using public ip, hence I am using this gateway yaml file
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: grafana-gateway
namespace: agung-ns
spec:
selector:
istio: ingressgateway # use Istio default gateway implementation
servers:
- port:
number: 15031
name: http-grafana
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grafana-global-route
namespace: agung-ns
spec:
hosts:
- "grafana.domain"
gateways:
- grafana-gateway
- mesh
http:
- route:
- destination:
host: "grafana.istio-system"
port:
number: 3000
weight: 100
I tried to curl it, but it returns 404 status, which means something wrong with routing logic and/or my configuration above.
curl -HHost:grafana.domain http://<my-istioingressgateway-publicip>:15031 -I
HTTP/1.1 503 Service Unavailable
date: Tue, 14 Aug 2018 13:04:27 GMT
server: envoy
transfer-encoding: chunked
Any idea?
I think the problem is that you refer service in different namespace. You need to add FQDN (grafana.istio-system.svc.cluster.local).
If you need istio, grafana, prometheus and jaeger integrated, exposed through gateway and with enabled security you can check the project I am working on:
https://github.com/kyma-project/kyma
I did expose it like this:
grafana.yml
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: grafana-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "my.dns.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grafana-vts
namespace: istio-system
spec:
hosts:
- "my.dns.com"
gateways:
- grafana-gateway
http:
- match:
- uri:
prefix: /
route:
- destination:
host: grafana
port:
number: 3000
then:
kubectl apply grafana.yml