I have two different clusters (EKS, v1.18) with their own meshes (v1.9.0).
I have a Thanos deployment on cluster A and a Prometheus deployment on cluster B (with the thanos sidecar running too). The goal is to have thanos query these sidecars in remote clusters to proxy queries to each cluster (block persistence using S3 or similar is out of scope for this issue) via an internal load balancer (ELB classic)
The resources for Gateway, Virtual Service and Service are in place in cluster B, and I can run Thanos locally when connected to the network and connect to the sidecars in cluster B successfully using gRPC.
The ServiceEntry for the FQDN from cluster B has been created in cluster A, resolution works, routing is correct, but the deployment in cluster A can't connect to cluster B.
Istio sidecars (from source workload, Thanos, in cluster A) show that the connection is being reset:
[2021-02-26T14:41:03.509Z] "POST /thanos.Store/Info HTTP/2" 0 - http2.remote_reset - "-" 5 0 4998 - "-" "grpc-go/1.29.1" "50912787-d528-994f-b8ad-78dd42081fea" "thanos.dev.integrations.internal.fqdn:10901" "-" - - 172.20.65.175:10901 172.30.9.174:37594 - default
I don't see the incoming request in cluster B's ingress gateway (I have a public one and a private one, I checked both just to be sure).
I have tried:
Forcing upgrade of http1.1 to http2 using DR
Forcing TLS to be disabled using DR
Excluding private LB CIDR range to bypass proxy
Resources (Cluster A)
ServiceEntry:
---
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: thanos-integrations-dev
namespace: thanos
spec:
hosts:
- thanos.dev.integrations.internal.fqdn
location: MESH_EXTERNAL
ports:
- name: grpc-thanos-int-dev
number: 10901
protocol: GRPC
resolution: DNS
Resources (Cluster B)
Gateway:
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
annotations:
meta.helm.sh/release-name: istio-routing-layer
meta.helm.sh/release-namespace: istio-system
creationTimestamp: "2021-02-25T11:37:49Z"
generation: 3
labels:
app.kubernetes.io/instance: istio-routing-layer
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: istio-routing-layer
app.kubernetes.io/version: 0.0.1
helm.sh/chart: istio-routing-layer-0.0.1
name: thanos
namespace: istio-system
spec:
selector:
istio: internal-ingressgateway
servers:
- hosts:
- thanos.dev.integrations.internal.fqdn
port:
name: grpc-thanos
number: 10901
VirtualService:
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
annotations:
meta.helm.sh/release-name: istio-routing-layer
meta.helm.sh/release-namespace: istio-system
creationTimestamp: "2021-02-25T11:37:49Z"
generation: 3
labels:
app.kubernetes.io/instance: istio-routing-layer
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: istio-routing-layer
app.kubernetes.io/version: 0.0.1
helm.sh/chart: istio-routing-layer-0.0.1
spec:
gateways:
- thanos
hosts:
- thanos.dev.integrations.internal.fqdn
http:
- route:
- destination:
host: thanos-sidecar.prometheus.svc.cluster.local
port:
number: 10901
Service:
---
apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: prometheus-thanos-istio
meta.helm.sh/release-namespace: prometheus
creationTimestamp: "2021-02-25T14:31:02Z"
labels:
app.kubernetes.io/instance: prometheus-thanos-istio
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: prometheus-thanos-istio
app.kubernetes.io/version: 0.0.1
helm.sh/chart: prometheus-thanos-istio-0.0.1
spec:
clusterIP: None
ports:
- name: grpc-thanos
port: 10901
protocol: TCP
targetPort: grpc
selector:
app: prometheus
component: server
sessionAffinity: None
type: ClusterIP
Related
I have installed the kube-prometheus stach from here and want to expose prometheus and grafana with a loadbalancer to get access to them from another cluster.
To acchieve this i have changed the prometheus-service.yaml by adding a type: LoadBalancer. When i try to access the exposed IP, the server says that the connection has timed out. What should i do to be able to access the prometheus server?
The altered prometheus-service.yaml looks like this:
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.38.0
name: prometheus-k8s
namespace: monitoring
spec:
ports:
- name: web
port: 9090
targetPort: web
- name: reloader-web
port: 8080
targetPort: reloader-web
type: LoadBalancer
selector:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
sessionAffinity: ClientIP
Ideas:
should I alter the networkpolicy to allow for external access? in that case, how?
I found a way to allow it to be exposed, it was networkpolicy. chekc link from github.
One has to add a seperate networkpolicy for prometheus to allow external, as shown here:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: prometheus-allow-external
namespace: monitoring
spec:
podSelector:
matchLabels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
ingress:
- ports:
- port: 9090
The problem is that i tought this was already done under the prometheus-networkPolicy.yaml:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.38.0
name: prometheus-k8s
namespace: monitoring
spec:
egress:
- {}
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- port: 9090
protocol: TCP
- port: 8080
protocol: TCP
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: grafana
ports:
- port: 9090
protocol: TCP
podSelector:
matchLabels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
policyTypes:
- Egress
- Ingress
Can anybody explain what the difference is?
Rancher ingress return 404 to service.
Setup: I have 6 VMs, one Rancher server x.x.x.51 (where dns domain.company is pointing to, TLS), and 5 VMs (one master and 4 worker x.x.x.52-56).
My service, gvm-gsad running in gvm namespace:
apiVersion: v1
kind: Service
metadata:
annotations:
field.cattle.io/publicEndpoints: "null"
meta.helm.sh/release-name: gvm
meta.helm.sh/release-namespace: gvm
creationTimestamp: "2021-11-15T21:14:21Z"
labels:
app.kubernetes.io/instance: gvm-gvm
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: gvm-gsad
app.kubernetes.io/version: "21.04"
helm.sh/chart: gvm-1.3.0
name: gvm-gsad
namespace: gvm
resourceVersion: "3488107"
uid: c1ddfdfa-3799-4945-841d-b6aa9a89f93a
spec:
clusterIP: 10.43.195.239
clusterIPs:
- 10.43.195.239
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: gsad
port: 80
protocol: TCP
targetPort: gsad
selector:
app.kubernetes.io/instance: gvm-gvm
app.kubernetes.io/name: gvm-gsad
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
My ingress configure: Ingress controller is default one from rancher.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
field.cattle.io/publicEndpoints: '[{"addresses":["172.16.12.53"],"port":443,"protocol":"HTTPS","serviceName":"gvm:gvm-gsad","ingressName":"gvm:gvm","hostname":"dtl.miproad.ad","path":"/gvm","allNodes":true}]'
creationTimestamp: "2021-11-16T19:22:45Z"
generation: 10
name: gvm
namespace: gvm
resourceVersion: "3508472"
uid: e99271a8-8553-45c8-b027-b259a453793c
spec:
rules:
- host: domain.company
http:
paths:
- backend:
service:
name: gvm-gsad
port:
number: 80
path: /gvm
pathType: Prefix
tls:
- hosts:
- domain.company
status:
loadBalancer:
ingress:
- ip: x.x.x.53
- ip: x.x.x.54
- ip: x.x.x.55
- ip: x.x.x.56
When i access it with https://domain.company/gvm then i get 404.
However, when i change the service to NodePort, i could access it with x.x.x.52:PORT normally. Meaning the deployment is running fine, just some configuration issue in ingress.
I checked this one: rancher 2.x thru ingress controller returns 404 but it does not help.
Thank you in advance!
Figured out the solution.
The domain.company is pointing to rancher (x.x.x.51). Where the ingress is running on (x.x.x.53,.54,.55,.56).
So, the solution is to create a new DNS called gvm.domain.company pointing to any ingress (x.x.x.53,.54,.55,.56) (you can have LoadBalancer here or use round robin DNS).
Then, the ingress definition is gvm.domain.company and path is "/".
Hope it helps others!
I have my AKS cluster where I am running my application without any problem. I have two deployments (backend-frontend) and I am using a service of type ClusterIP and an ingress controller to expose them publicly. I am trying to do the same with my Grafana deployment but it's not working. I am getting 404 error. If I go and execute the port forward I can access it over http://localhost:3000 and login successfully. The failure part is with ingress as it's created but with an empty address. I have three ingresses two of them shows the public IP in Address column and they are accessible through the URL while grafana url is not working.
Here is my Grafana Ingress:
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: loki-grafana-ingress
namespace: ingress-basic
labels:
app.kubernetes.io/instance: loki
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: grafana
app.kubernetes.io/version: 6.7.0
helm.sh/chart: grafana-5.7.10
annotations:
kubernetes.io/ingress.class: nginx
meta.helm.sh/release-name: loki
meta.helm.sh/release-namespace: ingress-basic
spec:
tls:
- hosts:
- xxx.xxx.me
secretName: sec-tls
rules:
- host: xxx.xxx.me
http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
serviceName: loki-grafana
servicePort: 80
status:
loadBalancer: {}
Attached are the Ingresses created:
Ingresses
Grafana Service
kind: Service
apiVersion: v1
metadata:
name: loki-grafana
namespace: ingress-basic
labels:
app.kubernetes.io/instance: loki
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: grafana
app.kubernetes.io/version: 6.7.0
helm.sh/chart: grafana-5.7.10
annotations:
meta.helm.sh/release-name: loki
meta.helm.sh/release-namespace: ingress-basic
spec:
ports:
- name: service
protocol: TCP
port: 80
targetPort: 3000
selector:
app.kubernetes.io/instance: loki
app.kubernetes.io/name: grafana
clusterIP: 10.0.189.19
type: ClusterIP
sessionAffinity: None
status:
loadBalancer: {}
When doing kubectl port-forward --namespace ingress-basic service/loki-grafana 3000:80
Grafana_localhost
Grana_URL
I created forward proxy server on EKS pods behind ALB(created by AWS Load Balancer Controller). All pod can take a response through 8118 port through ALB.
The resources like pod and ingress looked good to me. Then I tried if the proxy server work well with curl -Lx k8s-proxy-sample-domain.ap-uswest-1.elb.amazonaws.com:18118 ipinfo.io
Normally, I get random ip address from ipinfo.io. But it didn't.... So, I also did port-forad. Like this:
kubectl port-forward specifi-pod 8118:8118
Then I re-try redirect access on my host address.
curl -Lx localhost:8118 ipinfo.io
In this case, it went well. I cannot catch the reason. What's the difference between THROUGH ALB and port-forward. Should I use NLB for some reason? Or some misconfigure?
My environement
k8s version: v1.18.2
node type: fargate
Manifest
Here is my manifest.
---
apiVersion: v1
kind: Namespace
metadata:
name: tor-proxy
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: tor-proxy
name: tor-proxy-deployment
spec:
selector:
matchLabels:
app.kubernetes.io/name: tor-proxy
replicas: 5
template:
metadata:
labels:
app.kubernetes.io/name: tor-proxy
spec:
containers:
- image: dperson/torproxy
imagePullPolicy: Always
name: tor-proxy
ports:
- containerPort: 8118
---
apiVersion: v1
kind: Service
metadata:
labels:
name: tor-proxy
name: tor-proxy-service
namespace: tor-proxy
spec:
ports:
- port: 18118
targetPort: 8118
protocol: TCP
type: NodePort
selector:
app.kubernetes.io/name: tor-proxy
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: tor-proxy
name: tor-proxy-ingress
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 18118}]'
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: tor-proxy-service
servicePort: 18118
Use NLB not ALB, because it pass the client IP toward a target site through proxy server.
I am trying to set micro-services on Kubernetes on Google Cloud Platform. I've created a deployment, clusterIp and ingress configuration files.
First after creating a cluster, I run this command to install nginx ingress.
helm install my-nginx stable/nginx-ingress --set rbac.create=true
I use helm v3.
Then I apply deployment and clusterIp configurations.
deployment and clusterIp configurations:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-production-deployment
spec:
replicas: 2
selector:
matchLabels:
component: app-production
template:
metadata:
labels:
component: app-production
spec:
containers:
- name: app-production
image: eu.gcr.io/my-project/app:1.0
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: app-production-cluser-ip-service
spec:
type: ClusterIP
selector:
component: app-production
ports:
- port: 80
targetPort: 80
protocol: TCP
My ingress config is:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: app.example.com
http:
paths:
- path: /
backend:
serviceName: app-production-cluster-ip-service
servicePort: 80
I get this error from Google Cloud Platform logs withing ingress controller:
Error obtaining Endpoints for Service "default/app-production-cluster-ip-service": no object matching key "default/app-production-cluster-ip-service" in local store
But when I do kubectl get endpoints command the output is this:
NAME ENDPOINTS AGE
app-production-cluser-ip-service 10.60.0.12:80,10.60.1.13:80 17m
I am really not sure what I'm doing wrong.
The service name mentioned in the ingress not matching. Please recreate a service and check
apiVersion: v1
kind: Service
metadata:
name: app-production-cluster-ip-service
spec:
type: ClusterIP
selector:
component: app-production
ports:
- port: 80
targetPort: 80
protocol: TCP