strimzi 0.14: Kafka broker failed authentication - kubernetes

Question

I tried to build a Kafka cluster using Strimzi (0.14) in a Kubernetes cluster.
I use the examples come with the strimzi, i.e. examples/kafka/kafka-persistent.yaml.
This yaml file looks like this:
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 2.3.0
replicas: 3
listeners:
plain: {}
tls: {}
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.3"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 12Gi
deleteClaim: false
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 9Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}
kubectl apply -f examples/kafka/kafka-persistent.yaml
Both zookeepers and kafka brokers were brought up.
However, I saw errors in kafka broker logs:
[SocketServer brokerId=1] Failed authentication with /10.244.5.94 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-1-ListenerName(REPLICATION)-SSL-0]
Anyone know how to fix the problem?

Answer 1

One of the things which can cause this is if your cluster is using different DNS suffix for service domains (default is .cluster.local). You need to find out the right DNS suffix and use the environment variable KUBERNETES_SERVICE_DNS_DOMAIN in the Strimzi Cluster Operator deployment to override the default value.
If you exec into one of the Kafka or Zookeeper pods and do hostname -f it should show you the full hostname from which you can identifythe suffix.
(I pasted this from comments as a full answer since it helped to solve the question.)