How to resolve a zookeeper authentication failure when using Kafka with Kerberos - apache-kafka

Question

I am trying to use SASL between my kafka broker and zookeeper. When I start the kafka server
KAFKA_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka/config/kafka_server_jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf" \
./kafka-server-start.sh ../config/server.properties
I get the following error:
INFO TGT refresh thread started. (org.apache.zookeeper.Login)
DEBUG Client principal is "kafkabroker1/kafka.eigenroute.com#EIGENROUTE.COM". (org.apache.zookeeper.Login)
DEBUG Server principal is "krbtgt/EIGENROUTE.COM#EIGENROUTE.COM". (org.apache.zookeeper.Login)
INFO TGT valid starting at: Sat Dec 16 00:32:52 EST 2017 (org.apache.zookeeper.Login)
INFO TGT expires: Sat Dec 16 10:32:52 EST 2017 (org.apache.zookeeper.Login)
INFO TGT refresh sleeping until: Sat Dec 16 08:55:41 EST 2017 (org.apache.zookeeper.Login)
INFO Opening socket connection to server devel-2.sjml.com/173.243.38.81:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
DEBUG Closing ZkClient... (org.I0Itec.zkclient.ZkClient)
INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
DEBUG Closing ZooKeeper connected to zookeeper.eigenroute.com:2181 (org.I0Itec.zkclient.ZkConnection)
DEBUG Closing session: 0x0 (org.apache.zookeeper.ZooKeeper)
DEBUG Closing client for session: 0x0 (org.apache.zookeeper.ClientCnxn)
WARN Client session timed out, have not heard from server in 6004ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
DEBUG An exception was thrown while closing send thread for session 0x0 : Client session timed out, have not heard from server in 6004ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
DEBUG Ignoring exception during shutdown input (org.apache.zookeeper.ClientCnxnSocketNIO)
java.net.SocketException: Socket is not connected
My questions are: what is going on? And how to do I fix this?
Below are my configuration files. The first is server.properties:
# server.properties
broker.id=0
delete.topic.enable=true
listeners=SASL_PLAINTEXT://kafka.eigenroute.com:9092
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafkabroker1
inter.broker.listener.name=SASL_PLAINTEXT
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:kafkabroker1
...
zookeeper.connect=zookeeper.eigenroute.com:2181
zookeeper.connection.timeout.ms=6000
Here is kafka_server_jaas.conf
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
refreshKrb5Config=true
useKeyTab=true
storeKey=true
keyTab="/home/kafka/keytabs/kafka_broker1.keytab"
principal="kafkabroker1/kafka.eigenroute.com#EIGENROUTE.COM";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
refreshKrb5Config=true
useKeyTab=true
storeKey=true
keyTab="/home/kafka/keytabs/kafka_broker1.keytab"
principal="kafkabroker1/kafka.eigenroute.com#EIGENROUTE.COM";
};
This is the zookeeper file in /etc/init.d:
#!/bin/bash
export ZOOCFGDIR="/etc/zookeeper/conf/"
export SERVER_JVMFLAGS="-Djava.security.auth.login.config=/etc/zookeeper/conf/zookeeper.jaas -Djava.security.krb5.conf=/etc/krb5.conf"
echo "$#"
/usr/share/java/zookeeper-3.4.10/bin/zkServer.sh $# /etc/zookeeper/conf/zoo.cfg
Here is my zookeeper configuration file zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper
clientPort=2181
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000
I have enabled useTicketCache=true in the zookeeper.jaas file (does it belong there?):
Server {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
refreshKrb5Config=true
useKeyTab=true
keyTab="/home/kafka/keytabs/zookeeper.keytab"
storeKey=true
principal="zookeeper/zookeeper.eigenroute.com#EIGENROUTE.COM";
};
Finally, the keytab files are all readable:
$ ll /home/kafka/keytabs/
total 24
drwxr-xr-x 2 sjamal sjamal 4096 Dec 12 11:32 .
drwxr-xr-x 10 kafka kafka 4096 Dec 12 11:57 ..
-rw-r--r-- 1 root root 366 Dec 12 11:24 kafka_broker1.keytab
-rw-r--r-- 1 root root 426 Dec 12 11:31 testkafkaconsumer1.keytab
-rw-r--r-- 1 root root 426 Dec 12 11:31 testkafkaproducer1.keytab
-rw-r--r-- 1 root root 370 Dec 12 11:32 zookeeper.keytab
I have consulted the following resources:
apache-kafka-security-authorization-authentication-encryption.
Kafka SASL zookeeper authentication (this says to add the zookeeper.set.acl=true to server.properties, but this has no effect on the issue I am experiencing)
https://kafka.apache.org/documentation/#configuration (the official documentation, which could use improvement in some areas)
https://coheigea.blogspot.ca/2017/05/securing-apache-kafka-with-kerberos.html (a set of instructions that I initially started following)
Can someone suggest what the problem might be, and how to fix this? Thanks!
UPDATE: I ran netstat -tulnp | grep 2181 and ps aux | grep zookeeper, the output below shows that Zookeeper is running and listening on port 2181:
root#devel-2:~# netstat -tulnp | grep 2181
tcp6 0 0 :::2181 :::* LISTEN 3366/java
root#devel-2:~# ps aux | grep zookeeper
root 3366 0.0 0.6 3474796 26000 ? Sl Dec16 1:43 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/share/java/zookeeper-3.4.10/bin/../build/classes:/usr/share/java/zookeeper-3.4.10/bin/../build/lib/*.jar:/usr/share/java/zookeeper-3.4.10/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/share/java/zookeeper-3.4.10/bin/../lib/slf4j-api-1.6.1.jar:/usr/share/java/zookeeper-3.4.10/bin/../lib/netty-3.10.5.Final.jar:/usr/share/java/zookeeper-3.4.10/bin/../lib/log4j-1.2.16.jar:/usr/share/java/zookeeper-3.4.10/bin/../lib/jline-0.9.94.jar:/usr/share/java/zookeeper-3.4.10/bin/../zookeeper-3.4.10.jar:/usr/share/java/zookeeper-3.4.10/bin/../src/java/lib/*.jar:/etc/zookeeper/conf/: -Dsun.security.krb5.debug=true -Dlog4j.configuration=file:/etc/zookeeper/conf/log4j.properties -Djava.security.auth.login.config=/etc/zookeeper/conf/zookeeper.jaas -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/etc/zookeeper/conf/jaas.conf -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg
UPDATE: The software versions are:
Kerberos 5 version 1.12.1
Zookeeper 3.4.10
Kafka 0.11.0.0
UPDATE: Below is the content of my /etc/krb5.conf file:
[libdefaults]
default_realm = EIGENROUTE.COM
# The following krb5.conf variables are only for MIT Kerberos.
krb4_config = /etc/krb.conf
krb4_realms = /etc/krb.realms
kdc_timesync = 1
ccache_type = 4
forwardable = true
proxiable = true
# The following libdefaults parameters are only for Heimdal Kerberos.
v4_instance_resolve = false
v4_name_convert = {
host = {
rcmd = host
ftp = ftp
}
plain = {
something = something-else
}
}
fcc-mit-ticketflags = true
[realms]
EIGENROUTE.COM = {
kdc = krb.eigenroute.com
admin_server = krb.eigenroute.com
default_domain = eigenroute.com
}
ATHENA.MIT.EDU = {
kdc = kerberos.mit.edu:88
kdc = kerberos-1.mit.edu:88
kdc = kerberos-2.mit.edu:88
admin_server = kerberos.mit.edu
default_domain = mit.edu
}
MEDIA-LAB.MIT.EDU = {
kdc = kerberos.media.mit.edu
admin_server = kerberos.media.mit.edu
}
ZONE.MIT.EDU = {
kdc = casio.mit.edu
kdc = seiko.mit.edu
admin_server = casio.mit.edu
}
MOOF.MIT.EDU = {
kdc = three-headed-dogcow.mit.edu:88
kdc = three-headed-dogcow-1.mit.edu:88
admin_server = three-headed-dogcow.mit.edu
}
CSAIL.MIT.EDU = {
kdc = kerberos-1.csail.mit.edu
kdc = kerberos-2.csail.mit.edu
admin_server = kerberos.csail.mit.edu
default_domain = csail.mit.edu
krb524_server = krb524.csail.mit.edu
}
IHTFP.ORG = {
kdc = kerberos.ihtfp.org
admin_server = kerberos.ihtfp.org
}
GNU.ORG = {
kdc = kerberos.gnu.org
kdc = kerberos-2.gnu.org
kdc = kerberos-3.gnu.org
admin_server = kerberos.gnu.org
}
1TS.ORG = {
kdc = kerberos.1ts.org
admin_server = kerberos.1ts.org
}
GRATUITOUS.ORG = {
kdc = kerberos.gratuitous.org
admin_server = kerberos.gratuitous.org
}
DOOMCOM.ORG = {
kdc = kerberos.doomcom.org
admin_server = kerberos.doomcom.org
}
ANDREW.CMU.EDU = {
kdc = kerberos.andrew.cmu.edu
kdc = kerberos2.andrew.cmu.edu
kdc = kerberos3.andrew.cmu.edu
admin_server = kerberos.andrew.cmu.edu
default_domain = andrew.cmu.edu
}
CS.CMU.EDU = {
kdc = kerberos.cs.cmu.edu
kdc = kerberos-2.srv.cs.cmu.edu
admin_server = kerberos.cs.cmu.edu
}
DEMENTIA.ORG = {
kdc = kerberos.dementix.org
kdc = kerberos2.dementix.org
admin_server = kerberos.dementix.org
}
stanford.edu = {
kdc = krb5auth1.stanford.edu
kdc = krb5auth2.stanford.edu
kdc = krb5auth3.stanford.edu
master_kdc = krb5auth1.stanford.edu
admin_server = krb5-admin.stanford.edu
default_domain = stanford.edu
}
UTORONTO.CA = {
kdc = kerberos1.utoronto.ca
kdc = kerberos2.utoronto.ca
kdc = kerberos3.utoronto.ca
admin_server = kerberos1.utoronto.ca
default_domain = utoronto.ca
}
[domain_realm]
.mit.edu = ATHENA.MIT.EDU
mit.edu = ATHENA.MIT.EDU
.media.mit.edu = MEDIA-LAB.MIT.EDU
media.mit.edu = MEDIA-LAB.MIT.EDU
.csail.mit.edu = CSAIL.MIT.EDU
csail.mit.edu = CSAIL.MIT.EDU
.whoi.edu = ATHENA.MIT.EDU
whoi.edu = ATHENA.MIT.EDU
.stanford.edu = stanford.edu
.slac.stanford.edu = SLAC.STANFORD.EDU
.toronto.edu = UTORONTO.CA
.utoronto.ca = UTORONTO.CA
.eigenroute.com = EIGENROUTE.COM
eigenroute.com = EIGENROUTE.COM
[login]
krb4_convert = true
krb4_get_tickets = false
[logging]
kdc = FILE:/var/log/kerberos/krb5kdc.log
admin_server = FILE:/var/log/kerberos/kadmin.log
default = FILE:/var/log/kerberos/krb5lib.log