Where does Zookeeper(or Kafka) keep its ACL list?
When you run scripts like kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --list --topic test, where does Zookeeper (or Kafka) get its list?
I am trying to find a file that stores all the ACLs.
You can access Zookeeper using the zookeeper-shell.sh script.
There is a znode called kafka-acl where information about ACLs for group, topic, cluster and so on are stored.
You can list for example information about ACLs on topics ls /kafka-acl/Topic.
Then getting information about a specific topic with get /kafka-acl/Topic/test.
Since I landed here searching for the same information and eventually stubbled my way to the answer, I thought to add some additional information. Since Apache Kafka ver 2.0, for topics with patternType=PREFIXED, the acls are stored under a zookeeper node /kafka-acl-extended, this is in addition to the /kafka-acl node which hold topic details of patterType=LITERAL.
For more details, read - KAFKA-KIP-290
For additional reference :
If you look at your Zookeeper configuration file (zoo.cfg or zookeeper.properties), you will see the dataDir parameter, which tells you where the zookeeper stores its data.
For example,
dataDir=/tmp/confluent.iSAdMTvO/zookeeper/data
So, Kafka ACL list will be stored there, but in order to control it or view it, use
zookeeper-shell script. Because if you open the data to see, you won't be able to recognize it. See below for those who are curious.
Related
Is there any way to create kafka topic in kafka/zookeeper configuration files before I will run the services, so once they will start - the topics will be in place?
I have looked inside of script bin/kafka-topics.sh and found that in the end, it executes a live command on the live server. But since the server is here, its config files are here and zookeeper with its configs also are here, is it a way to predefined topics in advance?
Unfortunately haven't found any existing config keys for this.
The servers need to be running in order to allocate metadata and log directories for them, so no
On a couple of my clusters I'm seeing a discrepancy between the list of topics returned by zookeeper as compared to the broker i.e the following commands return different (fewer in the case of the broker) results
kafka-topics.sh --zookeeper $zookeeper --list
kafka-topics.sh --bootstrap-server $broker --command-config $clientProperties --list
I've seen this behaviour with multiple client versions which leads me to assume that the issue is on the server side, but I have no idea what the root cause is or how to fix it.
It causes an issue for me because I'm using some code that uses the brokers for GET operations like listing topics, and zookeeper for SET operations (create/updating topics). If the broker doesn't return a topic in a listing, then the code path leads to a CREATE action against zookeeper and that will be rejected (it will fail). Unfortunately, I don't control the code so I can't apply a fix there.
Nonetheless, surely the list of topics in zookeeper should be identical to the list in the broker?
I'm using Kafka (Amazon MSK) version 2.2.1
Thanks for the suggestions in this post. This is the explanation and solution:
The command "kafka-topics.sh --zookeeper" and "kafka-topics.sh --bootstrap-server" return two different outputs because the latter takes into account the configured ACLs which, in this case, prevent access to the topic metadata. Hence, the command through zookeeper provides the full list of topics, whereas the command through the broker provides only the topics for which ACLs are not configured.
In order to ensure the second command works as expected, you need to explicitly add to the ACL list of the affected topics access to the "DESCRIBE" operation
(^^ kudos to AWS Support for figuring this out)
I am trying to enable SASL username and password for a Kafka cluster with no ssl. I followed the steps on this Stackoverflow:
Kafka SASL zookeeper authentication
SASL authentication seems to be working for Kafka brokers. consumers and producers have to authenticate before writing to or reading from a topic. So far so good.
The problem is with creating and deleting topics on kafka. when I try to use the following command for example:
~/kafka/bin/kafka-topics.sh --list --zookeeper 10.x.y.z:2181
I am able to list all topics in the kafka cluster and create or delete any topic with no authentication at all.
I tried to follow the steps here:
Super User Authentication and Authorization
but nothing seem to work.
Any help in this matter is really appreciated.
Thanks & Regards,
Firas Khasawneh
You need to add zookeeper.set.acl=true to your Kafka server.properties so that Kafka will create everything in zookeeper with ACL set. For the topics which are already there, there will be no ACL and everyone can remove them directly from zookeeper.
Actually because of that mess, I had to delete everything from my zookeeper and Kafka and start from scratch.
But once everything is set, you can open zookeeper shell to verify that the ACL is indeed set:
KAFKA_OPTS="-Djava.security.auth.login.config=/path/to/your/jaas.conf" bin/zookeeper-shell.sh XXXXX:2181
From the shell you can run: getAcl /brokers/topics and check that not anyone from world have cdrwa
On a side note, the link you provided doesn't seem to reflect how the current version of Kafka stores information in zookeeper. I briefly looked at the codes and for those kafka-topics.sh commands, the topics information is from /brokers/topics instead of /config/topics
I was using a kafka topic, and it's metadata as well in my application. I hard deleted the topic from the zookeeper shell, by deleting the directories corresponding to that topic. After creating the topic again, I described the topic and found that no leaders have been assigned to this newly created topic. In the consumer, I can see repeated logs printing LEADER_NOT_AVAILABLE. Any reason as to what am I doing wrong? Or maybe is there a way to delete the metadata related to the kafka topic as well that I'm unaware of? Thanks in advance!
Deleting topics in Kafka hasn't been straightforward until recently. In general, you shouldn't attempt to delete Kafka topics by deleting metadata in Zookeeper. You should always use the included command line utilities.
First you need to make sure that deleting topics is enabled in the server.properties file on all brokers, and do a rolling restart if needed:
delete.topic.enable=true
After you restart the brokers to enable topic deletion, you should issue the delete command using the command line utilities:
./kafka-topics.sh —zookeeper <zookeeper_host>:2181 —delete —topic <topic_name>
If at this point, it's still stuck, try to run these two commands from the zookeeper shell to make sure and remove all metadata for that particular topic:
rmr /brokers/topics/<topic_name>
rmr /admin/delete_topics/<topic_name>
A few more details here:
https://medium.com/#contactsunny/manually-delete-apache-kafka-topics-424c7e016ff3
I've a 3-node unsecured kafka(v0.10.2.1) cluster with topic auto creation and deletion disabled with the following in server.properties
auto.create.topics.enable=false
delete.topic.enable=true
Topics are then created/altered on the cluster using bin/kafka-topics.sh. However, it looks like anyone can create topics on the cluster once they know the end points.
Is there a way to lock down topic creation/alteration to specific hosts to prevent abuses?
Edit 1:
Since ACL was suggested, I tried to restrict topic creation to select hosts using kafka-acls.sh.
I restarted the brokers after adding the following to server.properties, .
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=true
I tried the below to restrict topic creation on localhost.
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:* --cluster --operation Create --allow-host 127.0.0.1
However, I was still able to create topics from an other host using kafka-topics.sh with the right endpoints. Is it the case that ACLs can't be used without authentication?
You need to use access control lists (ACLs) to restrict such operations and that implies knowing who the caller is, so you need kafka to be secured by an authentication mechanism in the first place.
ACLs: http://kafka.apache.org/documentation.html#security_authz
Authentication can be done using SSL or SASL or by plugging in a custom provider, see the preceding sections of the same document.
Disabling auto-creation is not an access control mechanism, it only means that trying to produce to or consume from a topic will not create it automatically.