I'm trying to use Istio in a K8s 1.6 cluster on AWS.
I have a Kafka pod/service running the old fashion way, with a "kafka-zk-broker-kafka.dev" service without IP, so the kafka-zk-broker-kafka.dev service (I'm in the dev namespace) resolve to the internal name of my 3 Kafka pods. This is working great.
~ # nslookup kafka-zk-broker-kafka.dev
Name: kafka-zk-broker-kafka.dev
Address 1: 10.33.0.11 kafka-zk-kafka-0.kafka-zk-broker-kafka.dev.svc.cluster.local
Address 2: 10.38.96.16 kafka-zk-kafka-2.kafka-zk-broker-kafka.dev.svc.cluster.local
Address 3: 10.40.128.13 kafka-zk-kafka-1.kafka-zk-broker-kafka.dev.svc.cluster.local
I deployed a kafka producer application, using Istio sidecart as it is also exposing a gRPC port for internal uses.
Deployment went fine, but my application can't connect to to the "kafka-broker" service. DNS resolution is OK, but I can't reach the service port (TCP:9092) using either kafka client or telnet.
What I understand is that, when the Istio (envoy) sidecart is deployed, everything out of the POD is going through the Envoy proxy...
So the envoy proxy does not know how to reach regular services ?
Am I missing something ? is there a way to mix Istio/Envoy with regular k8s services ?
What you are doing should work, but I think you're running into this known bug: https://github.com/istio/issues/issues/37
Related
I know the below information is not enough to trace the issue but still, I want some solution.
We have Amazon EKS cluster.
Currently, we are facing the reachability of the Kafka pod issue.
Environment:
Total 10 nodes with Availability zone ap-south-1a,1b
I have a three replica of the Kafka cluster (Helm chart installation)
I have a three replica of the zookeeper (Helm chart installation)
Kafka using external advertised listener on port 19092
Kafka has service with an internal network load balancer
I have deployed a test-pod to check reachability of Kafka pod.
we are using cloud-map based DNS for advertized listener
Working:
When I run telnet command from ec2 like telnet 10.0.1.45 19092. It works as expected. IP 10.0.1.45 is a loadbalancer ip.
When I run telnet command from ec2 like telnet 10.0.1.69 31899. It works as expected. IP 10.0.1.69 is a actual node's ip and 31899 is nodeport.
Problem:
When I run same command from test-pod. like telnet 10.0.1.45 19092. It works sometime and sometime it will gives an error like telnet: Unable to connect to remote host: Connection timed out
The issue is something related to kube-proxy. we need help to resolve this issue.
Can anyone help to guide me?
Can I restart kube-proxy? Does it affect other pods/deployments?
I believe this problem is caused by AWS's NLB TCP-only nature (as mentioned in the comments).
In a nutshell, your pod-to-pod communication fails when hairpin is needed.
To confirm this is the root cause, you can verify that when the telnet works, kafka pod and client pod are not in the same EC2 node. And when they're in the same EC2 server, the telnet fails.
There are (at least) two approaches to tackle this issue:
Use K8s internal networking - Refer to k8s Service's URL
Every K8s service has its own DNS FQDN for internal usage (meaning using k8s network only, without reaching the LoadBalancer and come back to k8s again). You can just telnet this instead of the NodePort via the LB.
I.e. let's assume your kafka service is named kafka. Then you can just telnet kafka.svc.cluster.local (on the port exposed by kafka service)
Use K8s anti-affinity to make sure client and kafka are never scheduled in the same node.
Oh and as indicated in this answer you might need to make that service headless.
We have configured the Kubernetes cluster on bare-metal server with v1.15.1 and Istio-1.4.0 (demo) with mTLS enabled.
And our mysql server is outside the K8s cluster on Azure VM's.
Now when we inject istio-proxy while deploying the application we are unable to connect to mysql server via jdbc and also tried my mysql client. But when remove the istio-proxy by re-deploying we are able to connect instantly with out any issue.
When through many blogs wrt istio and mysql, tried with removing the default mesh policy but tht didnt work. The case in istio faq's is when the mysql is in k8s cluster with istio injected.
You can configure auto-mtls for istio by configuring values.global.mtls.auto=true (ie it uses mtls when possible and falls back for other connections
https://istio.io/docs/tasks/security/authentication/auto-mtls/
Serviceentry and destionation rule does the work form my case
I am using kubernetes 1.9.2 created but kubeadm.
this kubernetes cluster is running in 4 ec2 nodes.
I have a deployment that requires using cache in every pod.
in order to accomlish that we used session affinity from ClusterIP.
since I was ELB in front of my Kubernetes cluster I wonder how the session affinity is behaving.
the natural behavior would be that for every client IP a different will get the requests but given the traffic is transferred via ELB , whoch IP does the session affinity recognizes , the ELB IP or the actual Client IP?
when I check the traffic to the pods I see that 102 pods get all the requests and the 2 other pods are just waiting.
many thanks for any help.
SessionAffinity recognizes Client IP and ELB should pass the Client IP.
I think you should work with HTTP Headers and Classic Load Balancers and setup X-Forwarded-For: client-ip-address
Also, this seems to be a know issue enabling Session affinity goes to a single pod only #3056.
It was reported for 0.18.0 and 0.19.0 version of NGINX Ingress controller.
Issue was closed and commented that is was fixed in version 0.21.0, but in December initial author said it still doesn't work for him.
Is it possible to run PostgreSQL as a service inside an OpenShift cluster and get external traffic to it via an exposed route (the recommended default way for communicating from the outside)?
The OpenShift 3.9 documentation states this:
A router is configured to accept external requests and proxy them
based on the configured routes. This is limited to
HTTP/HTTPS(SNI)/TLS(SNI), which covers web applications.
PostgreSQL can to SSL and it can be configured to listen on port 443 (HTTPS), but I think it cannot do SNI yet. I would only run a single pod behind the service, so load balancing should not cause issues. If feasible, I would expect to connect into the cluster and to the service with psql -h ....
You can create a LoadBalancer service which will create a load balancer dedicated to just your pods, and this load balancer accepts TCP based traffic: https://docs.openshift.com/container-platform/3.9/admin_guide/tcp_ingress_external_ports.html#unique-external-ips-ingress-traffic-configure-service
So something like
oc expose dc postgres --type=LoadBalancer --name=postgres-lb
Would get you a publicly accessible PostgreSQL DB
I have come to the (preliminary) conclusion that OpenShift port forwarding (oc port-forward) offers the best way (well ... :) forward in this situation.
We wanted podnames to be resolved to IP's to configure the seed nodes in an akka cluster. This was happenning by using the concept of a headless service and stateful sets in Kubernetes. But, how do I expose a headless service externally to hit an endpoint from outside?
It is hard to expose a Kubernetes service to the outside, since this would require some complex TCP proxies. The reason for this is, that the headless services is only a DNS record with an IP for each pod. But these IPs are only reachable from within the cluster.
One solution is to expose this via Node ports, which means the ports are opened on the host itself. Unfortunately this makes the service discovery harder, because you don't know which host has a scheduled pod on it.
You can setup node ports via:
the services: https://kubernetes.io/docs/user-guide/services/#type-nodeport
or directly in the Pod by defining spec.containers[].ports[].hostPort
Another alternative is to use a LoadBalancer, if your cloud provider supports that. Unfortunately you cannot address each instance itself, since they share the same IP. This might not be suitable for your application.