Strimzi Kafka setup with GKE internal loadbalancer - apache-kafka

Question

Followed https://strimzi.io/quickstarts/ and https://strimzi.io/blog/2019/05/13/accessing-kafka-part-4/ to use GKE internal loadbalancer with Strimzi. After adding the internal load balancer Strimzi provisioned two loadbalancer service with external IP.
Kafka % k get svc -n kafka
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-kafka-0 LoadBalancer xx.xxx.xx.xxx bb.bb.bbb.bb 9094:30473/TCP 3d1h
my-cluster-kafka-bootstrap ClusterIP xx.xxx.xx.xxx <none> 9091/TCP,9092/TCP,9093/TCP 25d
my-cluster-kafka-brokers ClusterIP None <none> 9090/TCP,9091/TCP,9092/TCP,9093/TCP 25d
my-cluster-kafka-external-bootstrap LoadBalancer xx.xxx.xx.xxx aa.aa.aaa.aa 9094:30002/TCP 3d1h
my-cluster-zookeeper-client ClusterIP xx.xxx.xx.xxx <none> 2181/TCP 25d
my-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 25d
The producer/consumer flow is working inside the cluster using my-cluster-kafka-bootstrap and I can also curl the my-cluster-kafka-external-bootstrap addess aa.aa.aaa.aa:9094 from outside the cluster. However after producing to aa.aa.aaa.aa:9094 from outside the cluster my producer logged the error below.
Connection to node 0 (bb.bb.bbb.bb:9094) could not be established. Broker may not be available.
which seem to indicate my-cluster-kafka-external-bootstrap is forwarding the traffic to my-cluster-kafka-0. And per kubectl get svc -o yaml output only my-cluster-kafka-external-bootstrap was setup as a GKE internal LB. Since there are various firewall rules in our enviroment I suspect that my-cluster-kafka-0 needs to be set up as a GKE internal LB as well for the producer to work. Does this seem to be the issue? How do I update Strimzi to make both LB internal? Thanks.
A relevant question before Strimzi kafka accessing it privately with in GKE. But it didn't help after I turn off tls.

Answer 1

Answering own question. Appearntly Strimzi provision one LB per broker which is the my-cluster-kafka-0 here. The listener config can specify these per broker LBs like this https://strimzi.io/blog/2019/05/13/accessing-kafka-part-4/
# ...
listeners:
# ...
- name: external
port: 9094
type: loadbalancer
tls: true
authentication:
type: tls
configuration:
bootstrap:
annotations:
service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
brokers:
- broker: 0
annotations:
service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
- broker: 1
annotations:
service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
- broker: 2
annotations:
service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
# ...