KeeperErrorCode = InvalidACL when using kafka-configs - apache-kafka

Question

I'm trying to setup my kafka cluster to accept SASL_SSL / SCRAM authentication.
First of all, everything is working actually and clients connect using SASL_SSL / GSSAPI. My Zk servers are configured also for SASL authentication and TLS.
I'm using confluent docker images for Kafka and Zk:
confluentinc/cp-kafka:6.0.1
confluentinc/cp-zookeeper:5.5.3-3
So I just modified my setup to allow SCRAM-SHA-512 in Kafka: KAFKA_SASL_ENABLED_MECHANISMS=GSSAPI,SCRAM-SHA-512
Following these instructions I now want to add the users in Zk and this is where the problems start (from the Kafka node):
[root#kafka1 [RCI] ~]# /usr/bin/podman exec kafka kafka-configs --zk-tls-config-file /etc/kafka/secrets/zk-ssl.properties --zookeeper Zk:3181 --alter --entity-type topics --entity-name test_jerome --add-config 'retention.ms=1'
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
[2021-01-21 17:44:14,807] WARN zookeeper.ssl.keyStore.location not specified (org.apache.zookeeper.common.X509Util)
Error while executing config command with args '--zk-tls-config-file /etc/kafka/secrets/zk-ssl.properties --zookeeper Zk:3181 --alter --entity-type topics --entity-name test_jerome --add-config retention.ms=1'
org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /config/changes
at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:564)
at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1646)
at kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1568)
at kafka.zk.KafkaZkClient.createConfigChangeNotification(KafkaZkClient.scala:395)
at kafka.zk.AdminZkClient.changeEntityConfig(AdminZkClient.scala:385)
at kafka.zk.AdminZkClient.changeTopicConfig(AdminZkClient.scala:342)
at kafka.zk.AdminZkClient.changeConfigs(AdminZkClient.scala:278)
at kafka.admin.ConfigCommand$.alterConfigWithZk(ConfigCommand.scala:167)
at kafka.admin.ConfigCommand$.processCommandWithZk(ConfigCommand.scala:118)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:92)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Error: non zero exit code: 1: OCI runtime error
ZK logs are not really helpful:
[2021-01-21 17:58:08,333] INFO Successfully authenticated client: authenticationID=admin; authorizationID=admin. (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2021-01-21 17:58:08,333] INFO Successfully authenticated client: authenticationID=admin; authorizationID=admin. (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2021-01-21 17:58:08,333] INFO Setting authorizedID: admin (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2021-01-21 17:58:08,333] INFO Setting authorizedID: admin (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2021-01-21 17:58:08,334] INFO adding SASL authorization for authorizationID: admin (org.apache.zookeeper.server.ZooKeeperServer)
[2021-01-21 17:58:08,334] INFO adding SASL authorization for authorizationID: admin (org.apache.zookeeper.server.ZooKeeperServer)
As you can see, the SASL authentication works and the admin user is well identified.
If I login into the Zk shell (from the Zk node), you cann see that the acls are fully open:
getAcl /config/users
'world,'anyone
: cdrwa
getAcl /config
'world,'anyone
: cdrwa
getAcl /
'world,'anyone
: cdrwa
If I create the directory inside the Zk shell (from the Zk node) it works:
create /config/users/topicctl
Created /config/users/topicctl
There are no logs on the Zk server when I do this as I do not authenticate.
I now spend the afternoon on this problem without any progress.
What could be the problem please ?

Answer 1

I finally found my issue thanks to this post: Kafka not starting up if zookeeper.set.acl is set to true
I just added this info to the KAFKA_OPTS env variable: "-Dzookeeper.kerberos.removeHostFromPrincipal=true -Dzookeeper.kerberos.removeRealmFromPrincipal=true -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider -Dzookeeper.requireClientAuthScheme=sasl"
My main question is now: why sasl authentication was working before that !