So I have a Kafka cluster running with zookeeper with SSL. I gave a read permission to a user for a specific topic on the Kafka ACL: I can see it in zookeeper.
When this user is consuming the data, they are getting a Group Authorization Exception.
Do I need to add every group to the ACL? I am confuse about this error.
Thank you
You can update your post with exception trace.
Keeping that aside, the following is the exception we receive, if any client is not Authorized to perform Produce/Consume events.
EXCEPTION="org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [<<TopicName>>]\n"; EXCEPTION_TYPE="org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: <<Topic>>\n"
If you are receiving such exception, you need to make sure you have defined your ACL principle correctly.
Principle Definition
Kafka acls are defined in the general format of "Principal P is [Allowed/Denied] Operation O From Host H on any Resource R matching ResourcePattern RP".
In order to add, remove or list ACLs you can use the Kafka authorizer CLI. By default, if no ResourcePatterns match a specific Resource R, then R has no associated acls, and therefore no one other than super users is allowed to access R. If you want to change that behaviour, you can include the following in server.properties.
Sample Principle
Suppose you want to add an ACL "Principals User:Bob and User:Alice are allowed to perform Operation Read and Write on Topic Test-Topic from IP 198.51.100.0 and IP 198.51.100.1". You can do that by executing the CLI with following options:
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --allow-principal User:Alice --allow-host 198.51.100.0 --allow-host 198.51.100.1 --operation Read --operation Write --topic Test-topic
Related
How can authorize any consumer group to access a topic from a user that has permission in the ACL?
I am publishing data into topic test-1. And I authorized user-1 to have READ access to the Kafka ACL. But when I try to consume from the topic, I am getting a GROUP AUTHORIZATION EXCEPTION.
Is there a way to authorize any group on a topic for a particular user?
Yes you can authorize using wildcards so something like:
kafka-acls --bootstrap-server $host:$port --command-config adminclient-configs.conf --add --allow-principal \
User:$your_user --operation All --topic '$topic_name' --group '*'
You could use wildcard on topic also but not recommended and you can also adjust the operations more specifically(recommended), i.e - READ instead of using 'All' as in the example above.
I am trying to set up Kafka, where each user have several topics, but each topic may be consumed with any number of consumer group by the user the topic belongs to.
Kafka server version used: kafka_2.12-2.4.0 (Commit:77a89fcf8d7fa018)
Kafka client version used: confluent kafka 1.2.2
In Kafka ACL have successfully configured users, so they can only access only their own topic. I'm struggling to set up group permissions in such a way where each user can use any number of consumer groups just for their own topic without seeing what consumer groups others have.
The following enables every user to use any consumer group:
bin/kafka-acls.sh localhost:9092 --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=zookeeper.address --add --allow-principal User:* --operation Read --group '*'
However, according to https://docs.confluent.io/current/kafka/authorization.html Read operation implicitly grants Describe operation. As Describe operation includes access to 'ListGroup' API, which I do not want my users to be able to do, I executed the following:
bin/kafka-acls.sh localhost:9092 --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=zookeeper.address --add --deny-principal User:* --operation Describe --group '*'
The two commands above result in the following ACLs:
Current ACLs for resource `Group:LITERAL:*`:
User:* has Deny permission for operations: Describe from hosts: *
User:* has Allow permission for operations: Read from hosts: *
The problem with this is I'm getting the following exception:
Confluent.Kafka.ConsumeException: Broker: Group authorization failed
Which leads me to believe I'm either trying to achieve the impossible or trying it wrong.
TLDR: Is it possible to set up Kafka ACLs to allow using any consumer group without also granting ListGroups API permission at the same time?
Thanks for any answer.
For now decided to use prefix. Works well enough.
For those wondering how to do this:
bin/kafka-acls.sh localhost:9092 --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=zookeeper.address --add --allow-principal User:XYZ --operation Read --group 'ABC-' --resource-pattern-type prefixed
This piece of code will allow user 'XYZ' to use any consumer group starting with 'ABC-', like 'ABC-123'
I am collecting the data from different resources, each resource has one specific topic for each client.
I want to give the access for each user only to the corresponding topic, so they can't have access to all the topics.
I am working with Kafka 0.10 and I am using Kafka tools.
there is solution?
You need to configure Authorisation using ACL.
How to enable ACL:
In your server.properties file, you need to create an Authorizer by adding the following line:
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
Now you need to follow the docs in order to properly configure ACL based on your use cases.
Adding ACLs
Now once everything is in place, let's assume you have a topic called testTopic to which you want to grant read and write access only to user called Bob from a host with IP 197.5.6.1:
bin/kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 \
--add \
--allow-principal User:'Bob' --allow-host '197.5.6.1' \
--operation Read --operation Write \
--topic testTopic \
We have different consumer groups which consumes data from different topics.We have different partitions of the topic.We need to allow some consumer groups to have access to a particular topic so that only those groups can read from the topics.I was trying out Confluent Kafka Access control lists like this:
bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --consumer --topic test-topic --consumer-group Group-1
However as we understand it the ACLs can block access to Users and host addresses.Could we use ACLs to block consumer groups as well? If not is there any command that'll help me do it.
Thanks
ACLs on Consumer Groups isn't at all secure because any consumer can change their own group.id value. ACLs need to work off secure authentication credentials that cannot be spoofed or faked easily like X.509 Digital Certs or SASL/Kerberos credentials
I've a 3-node unsecured kafka(v0.10.2.1) cluster with topic auto creation and deletion disabled with the following in server.properties
auto.create.topics.enable=false
delete.topic.enable=true
Topics are then created/altered on the cluster using bin/kafka-topics.sh. However, it looks like anyone can create topics on the cluster once they know the end points.
Is there a way to lock down topic creation/alteration to specific hosts to prevent abuses?
Edit 1:
Since ACL was suggested, I tried to restrict topic creation to select hosts using kafka-acls.sh.
I restarted the brokers after adding the following to server.properties, .
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=true
I tried the below to restrict topic creation on localhost.
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:* --cluster --operation Create --allow-host 127.0.0.1
However, I was still able to create topics from an other host using kafka-topics.sh with the right endpoints. Is it the case that ACLs can't be used without authentication?
You need to use access control lists (ACLs) to restrict such operations and that implies knowing who the caller is, so you need kafka to be secured by an authentication mechanism in the first place.
ACLs: http://kafka.apache.org/documentation.html#security_authz
Authentication can be done using SSL or SASL or by plugging in a custom provider, see the preceding sections of the same document.
Disabling auto-creation is not an access control mechanism, it only means that trying to produce to or consume from a topic will not create it automatically.