I installed Prometheus using helm into Kubernets cluster (CentOS 8 VM) and want to access to dashboard from outside of cluster using VM IP
kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 27m
prometheus-grafana ClusterIP 10.98.154.200 <none> 80/TCP 27m
prometheus-kube-state-metrics ClusterIP 10.109.183.131 <none> 8080/TCP 27m
prometheus-operated ClusterIP None <none> 9090/TCP 27m
prometheus-prometheus-node-exporter ClusterIP 10.101.171.235 <none> 30206/TCP 27m
prometheus-prometheus-oper-alertmanager ClusterIP 10.109.205.136 <none> 9093/TCP 27m
prometheus-prometheus-oper-operator ClusterIP 10.111.243.35 <none> 8080/TCP,443/TCP 27m
prometheus-prometheus-oper-prometheus ClusterIP 10.106.76.22 <none> 9090/TCP 27m
i need to expose prometheus-prometheus-oper-prometheus service which works on port 9090 to be accessible from the outside on port 30000 using NodePort
http://Kubernetes_VM_IP:30000
so i created another service: but it fails services.yaml:
apiVersion: v1
kind: Service
metadata:
name: prometheus-service
namespace: monitoring
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '9090'
spec:
selector:
app: prometheus-operator-prometheus
type: NodePort
ports:
- port: 9090
nodePort: 30000
protocol: TCP
kubectl describe svc prometheus-prometheus-oper-prometheus -n monitoring
Name: prometheus-prometheus-oper-prometheus
Namespace: monitoring
Labels: app=prometheus-operator-prometheus
chart=prometheus-operator-8.12.2
heritage=Helm
release=prometheus
self-monitor=true
Annotations: <none>
Selector: app=prometheus,prometheus=prometheus-prometheus-oper-prometheus
Type: ClusterIP
IP: 10.106.76.22
Port: web 9090/TCP
TargetPort: 9090/TCP
Endpoints: 10.32.0.7:9090
Session Affinity: None
Events: <none>
Recreated prometheus and specified nodeport during installation:
helm install prometheus stable/prometheus-operator --namespace monitoring --set prometheus.service.nodePort=30000 --set prometheus.service.type=NodePort
For those using a values.yaml file, this is the correct structure :
prometheus:
service:
nodePort: 30000
type: NodePort
Related
I have k8s cluster v1.24.4+rke2r1, created by Rancher. I have already installed MetalLB, but when I try create pod with nginx and assign LoadBalancer I still have:
Service is ready. Load balancer is being provisioned
This is my pod and service config
apiVersion: v1
kind: Service
metadata:
name: nginx-service
annotations:
metallb.universe.tf/address-pool: public-ips
spec:
selector:
app: nginx
ports:
- port: 80
targetPort: 80
type: LoadBalancer
Pod
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
name: nginx
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.23.2-alpine
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 80
My MetalLB online config
configInline:
address-pools:
- addresses:
- 192.168.1.100-192.168.1.200
autoAssign: true
name: public-ips
protocol: layer2
When I describe my nginx-service I got
Name: nginx-service
Namespace: metal-test
Labels: <none>
Annotations: metallb.universe.tf/address-pool: public-ips
Selector: app=nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.127.38
IPs: 10.43.127.38
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 30010/TCP
Endpoints: 10.42.212.79:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
My service list
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
calico-system calico-kube-controllers-metrics ClusterIP 10.43.208.243 <none> 9094/TCP 21h
calico-system calico-typha ClusterIP 10.43.230.52 <none> 5473/TCP 21h
cattle-system cattle-cluster-agent ClusterIP 10.43.198.73 <none> 80/TCP,443/TCP 21h
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 21h
default nginx-service LoadBalancer 10.43.245.80 <pending> 8080:32146/TCP 66m
kube-system rke2-coredns-rke2-coredns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP 21h
kube-system rke2-metrics-server ClusterIP 10.43.84.128 <none> 443/TCP 21h
longhorn-system csi-attacher ClusterIP 10.43.49.124 <none> 12345/TCP 3h29m
longhorn-system csi-provisioner ClusterIP 10.43.236.132 <none> 12345/TCP 3h29m
longhorn-system csi-resizer ClusterIP 10.43.153.211 <none> 12345/TCP 3h29m
longhorn-system csi-snapshotter ClusterIP 10.43.182.109 <none> 12345/TCP 3h29m
longhorn-system longhorn-admission-webhook ClusterIP 10.43.49.242 <none> 9443/TCP 3h29m
longhorn-system longhorn-backend ClusterIP 10.43.71.124 <none> 9500/TCP 3h29m
longhorn-system longhorn-conversion-webhook ClusterIP 10.43.180.185 <none> 9443/TCP 3h29m
longhorn-system longhorn-engine-manager ClusterIP None <none> <none> 3h29m
longhorn-system longhorn-frontend ClusterIP 10.43.95.1 <none> 80/TCP 3h29m
longhorn-system longhorn-replica-manager ClusterIP None <none> <none> 3h29m
metallb metallb-webhook-service ClusterIP 10.43.211.242 <none> 443/TCP 178m
my metallb pod
NAME READY STATUS RESTARTS AGE
metallb-controller-6776dbc97d-kmkf9 1/1 Running 1 (177m ago) 177m
metallb-speaker-jrnmf 1/1 Running 0
I used this tutorial to install MetalLB http://xybernetics.com/techtalk/how-to-install-metallb-on-rancher-kubernetes-cluster/
I don't have any active firewall and I don't have nginx ingress on my cluster. Any idea what I do wrong? I do this on my local network.
I am new to Kubernetes and i am trying to deploy a MicroKubernetes cluster on 4 raspberry PIs.
I am struggling with setting up the dashboard since (no joke) a total of about 30 hours now and starting to be extremely frustrated .
I just cannot access the dashboard remotely.
Solutions that didnt work out:
No.1 Ingress:
I managed to enable ingress but it seems to be extremely complicated to connect it to the dashboard since i manually have to resolve DNS properties inside pods and host machines.
I eventually gave up on that. There is also no documentation whatsoever available how to set an ingress up without having a valid bought domain pointing at your Ingress Node.
If you are able to guide me through this, i am up for it.
No.2 Change service type of dashboard to LoadBalancer or NodePort:
With this method i can actually expose the dashboard... but it can only be accessed through https.... Since dashbaord seems to use self signed certificates or some other mechanism i cannot access the dashboard via a browser. The browsers(chrome firefox) always refuse to connect to the dashboard... When i try to access via http the browsers say i need to use https.
No.3 kube-proxy:
This only allows Localhost connections. YOu can pass arguments to kube proxy to allow other hosts to access the dashboard... but then again we have the https/http problem
At this point it is just amazing to me how extremly hard it is to just access this simple dashboard... Can anybody give any advice on how to access it ?
a#k8s-node-1:~/kubernetes$ kctl describe service kubernetes-dashboard -n kube-system
Name: kubernetes-dashboard
Namespace: kube-system
Labels: k8s-app=kubernetes-dashboard
Annotations: <none>
Selector: k8s-app=kubernetes-dashboard
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.152.183.249
IPs: 10.152.183.249
Port: <unset> 443/TCP
TargetPort: 8443/TCP
NodePort: <unset> 32228/TCP
Endpoints: 10.1.140.67:8443
Session Affinity: None
External Traffic Policy: Cluster
$ kubectl edit svc -n kube-system kubernetes-dashboard
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"k8s-app":"kubernetes-dashboard"},"name":"kubernetes-dashboard","namespace":"kube-system"},"spec":{"ports":[{"port":443,"targetPort":8443}],"selector":{"k8s>
creationTimestamp: "2022-03-21T14:30:10Z"
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
resourceVersion: "43060"
selfLink: /api/v1/namespaces/kube-system/services/kubernetes-dashboard
uid: fcb45ccc-070b-4a4d-b987-41f5b7777559
spec:
clusterIP: 10.152.183.249
clusterIPs:
- 10.152.183.249
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- nodePort: 32228
port: 443
protocol: TCP
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
a#k8s-node-1:~/kubernetes$ kctl get services -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
metrics-server ClusterIP 10.152.183.233 <none> 443/TCP 165m
kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 142m
dashboard-metrics-scraper ClusterIP 10.152.183.202 <none> 8000/TCP 32m
kubernetes-dashboard NodePort 10.152.183.249 <none> 443:32228/TCP 32m
a#k8s-node-1:~/kubernetes$ cat dashboard-ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
name: dashboard
namespace: kube-system
spec:
rules:
- host: nonexistent.net
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kubernetes-dashboard
port:
number: 8080
a#k8s-node-1:~/kubernetes$ kctl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system calico-node-c4shb 1/1 Running 0 3h23m 192.168.180.47 k8s-node-2 <none> <none>
ingress nginx-ingress-microk8s-controller-nvcvx 1/1 Running 0 3h12m 10.1.140.66 k8s-node-2 <none> <none>
kube-system calico-node-ptwmk 1/1 Running 0 3h23m 192.168.180.48 k8s-node-3 <none> <none>
ingress nginx-ingress-microk8s-controller-hksg7 1/1 Running 0 3h12m 10.1.55.131 k8s-node-4 <none> <none>
ingress nginx-ingress-microk8s-controller-tk9dj 1/1 Running 0 3h12m 10.1.76.129 k8s-node-3 <none> <none>
ingress nginx-ingress-microk8s-controller-c8t54 1/1 Running 0 3h12m 10.1.109.66 k8s-node-1 <none> <none>
kube-system calico-node-k65fz 1/1 Running 0 3h22m 192.168.180.52 k8s-node-4 <none> <none>
kube-system coredns-64c6478b6c-584s8 1/1 Running 0 177m 10.1.109.67 k8s-node-1 <none> <none>
kube-system calico-kube-controllers-6966456d6b-vvnm6 1/1 Running 0 3h24m 10.1.109.65 k8s-node-1 <none> <none>
kube-system calico-node-7jhz9 1/1 Running 0 3h33m 192.168.180.46 k8s-node-1 <none> <none>
kube-system metrics-server-647bdc584d-ldf8q 1/1 Running 1 (3h19m ago) 3h20m 10.1.55.129 k8s-node-4 <none> <none>
kube-system kubernetes-dashboard-585bdb5648-8s9xt 1/1 Running 0 67m 10.1.140.67 k8s-node-2 <none> <none>
kube-system dashboard-metrics-scraper-69d9497b54-x7vt9 1/1 Running 0 67m 10.1.55.132 k8s-node-4 <none> <none>
Using an ingress is indeed the preferred way, but since you seem to have trouble in your environment, you can indeed use a LoadBalancer service.
To avoid the problem with the automatically generated certificates, provide your certificate and private key to the dashboard, for example as a secret, and use the flags --tls-key-file and --tls-cert-file to point to the certificate. More details: https://github.com/kubernetes/dashboard/blob/master/docs/user/certificate-management.md
Hey it's been quite days struggling to make the sample book app running. I am new to istio and trying to get understand it. I followed this demo of an other way of setting up the bookinfo. I am using minikube in a virtualbox machine with docker as a driver. I set metalLB as a loadBalancer for ingress-gateway, here is the configmap i used for metalLB :
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: custom-ip-space
protocol: layer2
addresses:
- 192.168.49.2/28
the 192.168.49.2 is the result of the command: minikube ip
The ingressgateway yaml file:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: bookinfo-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bookinfo
spec:
hosts:
- "*"
gateways:
- bookinfo-gateway
http:
- route:
- destination:
host: productpage
port:
number: 9080
and the output command of kubectl get svc -n istio-system:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana ClusterIP 10.111.105.179 <none> 3000/TCP 34m
istio-citadel ClusterIP 10.100.38.218 <none> 8060/TCP,15014/TCP 34m
istio-egressgateway ClusterIP 10.101.66.207 <none> 80/TCP,443/TCP,15443/TCP 34m
istio-galley ClusterIP 10.103.112.155 <none> 443/TCP,15014/TCP,9901/TCP 34m
istio-ingressgateway LoadBalancer 10.97.23.39 192.168.49.0 15020:32717/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:32199/TCP,15030:30010/TCP,15031:30189/TCP,15032:31134/TCP,15443:30748/TCP 34m
istio-pilot ClusterIP 10.108.133.31 <none> 15010/TCP,15011/TCP,8080/TCP,15014/TCP 34m
istio-policy ClusterIP 10.100.74.207 <none> 9091/TCP,15004/TCP,15014/TCP 34m
istio-sidecar-injector ClusterIP 10.97.224.99 <none> 443/TCP,15014/TCP 34m
istio-telemetry ClusterIP 10.101.165.139 <none> 9091/TCP,15004/TCP,15014/TCP,42422/TCP 34m
jaeger-agent ClusterIP None <none> 5775/UDP,6831/UDP,6832/UDP 34m
jaeger-collector ClusterIP 10.111.188.83 <none> 14267/TCP,14268/TCP,14250/TCP 34m
jaeger-query ClusterIP 10.103.148.144 <none> 16686/TCP 34m
kiali ClusterIP 10.111.57.222 <none> 20001/TCP 34m
prometheus ClusterIP 10.107.204.95 <none> 9090/TCP 34m
tracing ClusterIP 10.104.88.173 <none> 80/TCP 34m
zipkin ClusterIP 10.111.162.93 <none> 9411/TCP 34m
and when trying to curl 192.168.49.0:80/productpage I am getting :
* Trying 192.168.49.0...
* TCP_NODELAY set
* Immediate connect fail for 192.168.49.0: Network is unreachable
* Closing connection 0
curl: (7) Couldn't connect to server
myhost#k8s:~$ curl 192.168.49.0:80/productpage
curl: (7) Couldn't connect to server
and before setting up the metalLB, I was getting connection refused!
Any solution for this please ? as it's been 5 days struggling to fix it.
I followed the steps here and all steps are ok!
In my opinion, this is a problem with the MetalLB configuration.
You are trying to give MetalLB control over IPs from the 192.168.49.2/28 network.
We can calculate for 192.168.49.2/28 network: HostMin=192.168.49.1 and HostMax=192.168.49.14.
As we can see, your istio-ingressgateway LoadBalancer Service is assigned the address 192.168.49.0 and I think that is the cause of the problem.
I recommend changing from 192.168.49.2/28 to a range, such as 192.168.49.10-192.168.49.20.
I've created an example to illustrate you how your configuration can be changed.
As you can see, at the beginning I had the configuration exactly like you (I also couldn't connect to the server using the curl command):
$ kubectl get svc -n istio-system istio-ingressgateway
NAME TYPE CLUSTER-IP EXTERNAL-IP
istio-ingressgateway LoadBalancer 10.109.75.19 192.168.49.0
$ curl 192.168.49.0:80/productpage
curl: (7) Couldn't connect to server
First, I modified the config ConfigMap:
NOTE: I changed 192.168.49.2/28 to 192.168.49.10-192.168.49.20
$ kubectl edit cm config -n metallb-system
Then I restarted all the controller and speaker Pods to force MetalLB to use new config (see: Metallb ConfigMap update).
$ kubectl delete pod -n metallb-system --all
pod "controller-65db86ddc6-gf49h" deleted
pod "speaker-7l66v" deleted
After some time, we should see a new EXTERNAL-IP assigned to the istio-ingressgateway Service:
kubectl get svc -n istio-system istio-ingressgateway
NAME TYPE CLUSTER-IP EXTERNAL-IP AGE
istio-ingressgateway LoadBalancer 10.106.170.227 192.168.49.10
Finally, we can check if it works as expected:
$ curl 192.168.49.10:80/productpage
<!DOCTYPE html>
<html>
<head>
<title>Simple Bookstore App</title>
...
I have a simple k3s cluster with the Ambassador ingress controller installed as per the docs
When I try to access the service through my browser, I just get a "no healthy upstream" message.
These are my configs:
$ kubectl describe svc web-test-service
Name: web-test-service
Namespace: default
Labels: app=web-test
Annotations: Selector: app=web-test
Type: ClusterIP
IP: 10.43.109.123
Port: <unset> 8080/TCP
TargetPort: 8080/TCP
Endpoints: 10.42.1.19:8080
Session Affinity: None
Events: <none>
$ kubectl describe svc ambassador
Name: ambassador
Namespace: default
Labels: app.kubernetes.io/component=ambassador-service
Annotations: Selector: service=ambassador
Type: LoadBalancer
IP: 10.43.12.194
LoadBalancer Ingress: 10.136.64.114
Port: <unset> 80/TCP
TargetPort: 8080/TCP
NodePort: <unset> 30005/TCP
Endpoints: 10.42.0.10:8080,10.42.1.28:8080,10.42.1.29:8080
Session Affinity: None
External Traffic Policy: Local
HealthCheck NodePort: 30928
Events: <none>
$ kubectl get po
NAME READY STATUS RESTARTS AGE
web-test-5594bffd47-8pzdk 1/1 Running 0 175m
svclb-ambassador-p5rr7 1/1 Running 0 24m
svclb-ambassador-k4j52 1/1 Running 0 24m
ambassador-58b444b8-tqjkk 1/1 Running 0 24m
ambassador-58b444b8-b9x7v 1/1 Running 0 24m
ambassador-58b444b8-wfclj 1/1 Running 0 24m
I've checked the service logs and the application is up and running and listening on port 8080.
I'm configuring Istio using Helm. Here you can find my istio-config.yaml:
global:
proxy:
accessLogFile: "/dev/stdout"
resources:
requests:
cpu: 10m
memory: 40Mi
disablePolicyChecks: false
sidecarInjectorWebhook:
enabled: true
rewriteAppHTTPProbe: false
pilot:
autoscaleEnabled: false
traceSampling: 100.0
resources:
requests:
cpu: 10m
memory: 100Mi
mixer:
policy:
enabled: true
autoscaleEnabled: false
resources:
requests:
cpu: 10m
memory: 100Mi
telemetry:
enabled: true
autoscaleEnabled: false
resources:
requests:
cpu: 50m
memory: 100Mi
adapters:
stdio:
enabled: true
grafana:
enabled: true
tracing:
enabled: true
kiali:
enabled: true
createDemoSecret: true
gateways:
istio-ingressgateway:
autoscaleEnabled: false
resources:
requests:
cpu: 10m
memory: 40Mi
istio-egressgateway:
enabled: true
autoscaleEnabled: false
resources:
requests:
cpu: 10m
memory: 40Mi
global:
controlPlaneSecurityEnabled: false
mtls:
enabled: false
Then I deployed a bunch of microservices using istioctl, all of them are simple REST call using HTTP. They can communicate with each other without any issue. If I exposed them with NodePorts I can reach and communicate with them correctly.
Here are my Services:
$ kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default activemq ClusterIP None <none> 61616/TCP 3h17m
default activemq-np NodePort 10.110.76.147 <none> 8161:30061/TCP 3h17m
default api-exchange ClusterIP None <none> 8080/TCP 3h16m
default api-response ClusterIP None <none> 8080/TCP 3h16m
default authorization-server ClusterIP None <none> 8080/TCP 3h17m
default de-communication ClusterIP None <none> 8080/TCP 3h16m
default gateway ClusterIP None <none> 8080/TCP 3h17m
default gateway-np NodePort 10.96.123.57 <none> 8080:30080/TCP 3h17m
default identity ClusterIP None <none> 88/TCP,8080/TCP 3h18m
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h19m
default matchengine ClusterIP None <none> 8080/TCP 3h16m
default monitor-redis ClusterIP None <none> 8081/TCP 3h17m
default monitor-redis-np NodePort 10.106.178.13 <none> 8081:30082/TCP 3h17m
default postgres ClusterIP None <none> 5432/TCP 3h18m
default postgres-np NodePort 10.106.223.216 <none> 5432:30032/TCP 3h18m
default redis ClusterIP None <none> 6379/TCP 3h18m
default redis-np NodePort 10.101.167.194 <none> 6379:30079/TCP 3h18m
default synchronization ClusterIP None <none> 8080/TCP 3h15m
default tx-flow ClusterIP None <none> 8080/TCP 3h15m
default tx-manager ClusterIP None <none> 8080/TCP 3h15m
default tx-scheduler ClusterIP None <none> 8080/TCP 3h15m
default ubc-config ClusterIP None <none> 8080/TCP 3h16m
default ubc-services-config ClusterIP None <none> 8888/TCP 3h18m
default ubc-services-config-np NodePort 10.110.11.213 <none> 8888:30088/TCP 3h18m
default user-admin ClusterIP None <none> 8080/TCP 3h17m
default web-exchange-np NodePort 10.105.244.194 <none> 80:30081/TCP 3h15m
istio-system grafana ClusterIP 10.97.134.230 <none> 3000/TCP 3h22m
istio-system istio-citadel ClusterIP 10.99.159.56 <none> 8060/TCP,15014/TCP 3h22m
istio-system istio-egressgateway ClusterIP 10.97.71.204 <none> 80/TCP,443/TCP,15443/TCP 3h22m
istio-system istio-galley ClusterIP 10.98.111.27 <none> 443/TCP,15014/TCP,9901/TCP 3h22m
istio-system istio-ingressgateway LoadBalancer 10.96.182.202 <pending> 15020:30936/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31913/TCP,15030:30606/TCP,15031:32127/TCP,15032:30362/TCP,15443:31416/TCP 3h22m
istio-system istio-pilot ClusterIP 10.101.117.169 <none> 15010/TCP,15011/TCP,8080/TCP,15014/TCP 3h22m
istio-system istio-policy ClusterIP 10.97.247.54 <none> 9091/TCP,15004/TCP,15014/TCP 3h22m
istio-system istio-sidecar-injector ClusterIP 10.101.219.141 <none> 443/TCP 3h22m
istio-system istio-telemetry ClusterIP 10.109.108.78 <none> 9091/TCP,15004/TCP,15014/TCP,42422/TCP 3h22m
istio-system jaeger-agent ClusterIP None <none> 5775/UDP,6831/UDP,6832/UDP 3h22m
istio-system jaeger-collector ClusterIP 10.97.255.231 <none> 14267/TCP,14268/TCP 3h22m
istio-system jaeger-query ClusterIP 10.104.80.162 <none> 16686/TCP 3h22m
istio-system kiali ClusterIP 10.104.41.71 <none> 20001/TCP 3h22m
istio-system kiali-np NodePort 10.100.99.141 <none> 20001:30085/TCP 29h
istio-system prometheus ClusterIP 10.110.46.60 <none> 9090/TCP 3h22m
istio-system tracing ClusterIP 10.111.173.205 <none> 80/TCP 3h22m
istio-system zipkin ClusterIP 10.101.144.199 <none> 9411/TCP 3h22m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 54d
kube-system tiller-deploy ClusterIP 10.105.162.195 <none> 44134/TCP 24d
I created an ingress gateway and one VirtualService to route calls from outside the cluster. Here are my gateway and Virtual Services configurations:
Gateway:
$ kubectl describe gateway iris-gateway
Name: iris-gateway
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"iris-gateway","namespace":"default"},"s...
API Version: networking.istio.io/v1alpha3
Kind: Gateway
Metadata:
Creation Timestamp: 2019-08-23T17:25:20Z
Generation: 1
Resource Version: 7093263
Self Link: /apis/networking.istio.io/v1alpha3/namespaces/default/gateways/iris-gateway
UID: 4c4fac7d-a698-4c9c-97e6-ebc7416c96a8
Spec:
Selector:
Istio: ingressgateway
Servers:
Hosts:
*
Port:
Name: http
Number: 80
Protocol: HTTP
Events: <none>
Virtual Services:
$ kubectl describe virtualservice apiexg
Name: apiexg
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"apiexg","namespace":"default"},"...
API Version: networking.istio.io/v1alpha3
Kind: VirtualService
Metadata:
Creation Timestamp: 2019-08-23T19:26:16Z
Generation: 1
Resource Version: 7107510
Self Link: /apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/apiexg
UID: 861bca0d-be98-4bfb-bf92-b2bd2f1b703f
Spec:
Gateways:
iris-gateway
Hosts:
*
Http:
Match:
Uri:
Prefix: /api-exchange
Route:
Destination:
Host: api-exchange.default.svc.cluster.local
Port:
Number: 8080
Events: <none>
When I make a call to the service I always got a 503 Service Unavailable:
curl -X POST http://172.30.7.129:31380/api-exchange/ -vvv
* About to connect() to 172.30.7.129 port 31380 (#0)
* Trying 172.30.7.129...
* Connected to 172.30.7.129 (172.30.7.129) port 31380 (#0)
> POST /api-exchange/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 172.30.7.129:31380
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-length: 19
< content-type: text/plain
< date: Fri, 23 Aug 2019 21:49:33 GMT
< server: istio-envoy
<
* Connection #0 to host 172.30.7.129 left intact
no healthy upstream
Here is log output for istio-ingressgateway pod:
[2019-08-23 21:49:34.185][38][warning][upstream] [external/envoy/source/common/upstream/original_dst_cluster.cc:110] original_dst_load_balancer: No downstream connection or no original_dst.
Versions:
$ istioctl version --remote
client version: 1.2.4
citadel version: 1.2.4
egressgateway version: 94746ccd404a8e056483dd02e4e478097b950da6-dirty
galley version: 1.2.4
ingressgateway version: 94746ccd404a8e056483dd02e4e478097b950da6-dirty
pilot version: 1.2.4
policy version: 1.2.4
sidecar-injector version: 1.2.4
telemetry version: 1.2.4
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:18:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Istio installation:
$ helm install /opt/istio-1.2.4/install/kubernetes/helm/istio-init --name istio-init --namespace istio-system
$ helm install /opt/istio-1.2.4/install/kubernetes/helm/istio --name istio --namespace istio-system --values istio-config/istio-config.yaml
Environment:
I did the same configuration over a Oracle Virtual Appliance Virtual Server with RHEL 7 and over a cluster of 3 Physical Servers with RHEL 7.
I solve this problem. istio-gateway was not capable to do redirect due to one of my services have a ClusterIP assigned:
$ kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default activemq ClusterIP None <none> 61616/TCP 3h17m
default api-exchange ClusterIP None <none> 8080/TCP 3h16m
default api-response ClusterIP None <none> 8080/TCP 3h16m
default authorization-server ClusterIP None <none> 8080/TCP 3h17m
default de-communication ClusterIP None <none> 8080/TCP 3h16m
default gateway ClusterIP None <none> 8080/TCP 3h17m
default identity ClusterIP None <none> 88/TCP,8080/TCP 3h18m
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h19m
default matchengine ClusterIP None <none> 8080/TCP 3h16m
default monitor-redis ClusterIP None <none> 8081/TCP 3h17m
default postgres ClusterIP None <none> 5432/TCP 3h18m
default redis ClusterIP None <none> 6379/TCP 3h18m
default synchronization ClusterIP None <none> 8080/TCP 3h15m
default tx-flow ClusterIP None <none> 8080/TCP 3h15m
default tx-manager ClusterIP None <none> 8080/TCP 3h15m
default tx-scheduler ClusterIP None <none> 8080/TCP 3h15m
default ubc-config ClusterIP None <none> 8080/TCP 3h16m
default ubc-services-config ClusterIP None <none> 8888/TCP 3h18m
default user-admin ClusterIP None <none> 8080/TCP 3h17m
Here one of my YAML with ClusterIP: None:
apiVersion: v1
kind: Service
metadata:
name: ubc-config
labels:
app: ubc-config
spec:
clusterIP: None
ports:
- port: 8080
name: ubc-config
selector:
app: ubc-config
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ubc-config
spec:
selector:
matchLabels:
app: ubc-config
replicas: 1
template:
metadata:
labels:
app: ubc-config
spec:
containers:
- name: ubc-config
image: ubc-config
ports:
- containerPort: 8080
As you can see, Service.spec.ClusterIP is set to NONE. To solve the problem I only change my YAML configuration to:
apiVersion: v1
kind: Service
metadata:
name: ubc-config
labels:
app: ubc-config
spec:
ports:
- port: 8080
name: http-ubcconfig
selector:
app: ubc-config
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ubc-config
spec:
selector:
matchLabels:
app: ubc-config
replicas: 1
template:
metadata:
labels:
app: ubc-config
spec:
containers:
- name: ubc-config
image: ubc-config
ports:
- containerPort: 8080
name: http-ubcconfig
I hope this helps someone.