How can authorize any consumer group to access a topic from a user that has permission in the ACL?
I am publishing data into topic test-1. And I authorized user-1 to have READ access to the Kafka ACL. But when I try to consume from the topic, I am getting a GROUP AUTHORIZATION EXCEPTION.
Is there a way to authorize any group on a topic for a particular user?
Yes you can authorize using wildcards so something like:
kafka-acls --bootstrap-server $host:$port --command-config adminclient-configs.conf --add --allow-principal \
User:$your_user --operation All --topic '$topic_name' --group '*'
You could use wildcard on topic also but not recommended and you can also adjust the operations more specifically(recommended), i.e - READ instead of using 'All' as in the example above.
Related
So I have a Kafka cluster running with zookeeper with SSL. I gave a read permission to a user for a specific topic on the Kafka ACL: I can see it in zookeeper.
When this user is consuming the data, they are getting a Group Authorization Exception.
Do I need to add every group to the ACL? I am confuse about this error.
Thank you
You can update your post with exception trace.
Keeping that aside, the following is the exception we receive, if any client is not Authorized to perform Produce/Consume events.
EXCEPTION="org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [<<TopicName>>]\n"; EXCEPTION_TYPE="org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: <<Topic>>\n"
If you are receiving such exception, you need to make sure you have defined your ACL principle correctly.
Principle Definition
Kafka acls are defined in the general format of "Principal P is [Allowed/Denied] Operation O From Host H on any Resource R matching ResourcePattern RP".
In order to add, remove or list ACLs you can use the Kafka authorizer CLI. By default, if no ResourcePatterns match a specific Resource R, then R has no associated acls, and therefore no one other than super users is allowed to access R. If you want to change that behaviour, you can include the following in server.properties.
Sample Principle
Suppose you want to add an ACL "Principals User:Bob and User:Alice are allowed to perform Operation Read and Write on Topic Test-Topic from IP 198.51.100.0 and IP 198.51.100.1". You can do that by executing the CLI with following options:
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --allow-principal User:Alice --allow-host 198.51.100.0 --allow-host 198.51.100.1 --operation Read --operation Write --topic Test-topic
I want to delete a Kafka user.
Is this something that can be achieved?
Honestly, im not even sure im actually understanding the concept of Kafka users.
I started by reading this, but im not being able to easily extract any conclusion.
Thank you
Kafka user per say it's just the sum of its permissions, there is no "real" kafka user object in kafka.
Let's say that the "kafka" saves just the definitions of authorisations the user/principle possess.
You can remove the principle permissions, with an adequate command, for example kafka-acls.sh with the --remove parameter.
You can remove SCRAM password and/or quota config from kafka with a command like:
kafka-configs.sh --bootstrap-server 127.0.0.1:9092 --alter --delete-config "SCRAM-SHA-256" --entity-type users --entity-name USERNAME
And describe existing user configs with:
kafka-configs.sh --bootstrap-server 127.0.0.1:9092 --describe --entity-type users
I am trying to set up Kafka, where each user have several topics, but each topic may be consumed with any number of consumer group by the user the topic belongs to.
Kafka server version used: kafka_2.12-2.4.0 (Commit:77a89fcf8d7fa018)
Kafka client version used: confluent kafka 1.2.2
In Kafka ACL have successfully configured users, so they can only access only their own topic. I'm struggling to set up group permissions in such a way where each user can use any number of consumer groups just for their own topic without seeing what consumer groups others have.
The following enables every user to use any consumer group:
bin/kafka-acls.sh localhost:9092 --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=zookeeper.address --add --allow-principal User:* --operation Read --group '*'
However, according to https://docs.confluent.io/current/kafka/authorization.html Read operation implicitly grants Describe operation. As Describe operation includes access to 'ListGroup' API, which I do not want my users to be able to do, I executed the following:
bin/kafka-acls.sh localhost:9092 --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=zookeeper.address --add --deny-principal User:* --operation Describe --group '*'
The two commands above result in the following ACLs:
Current ACLs for resource `Group:LITERAL:*`:
User:* has Deny permission for operations: Describe from hosts: *
User:* has Allow permission for operations: Read from hosts: *
The problem with this is I'm getting the following exception:
Confluent.Kafka.ConsumeException: Broker: Group authorization failed
Which leads me to believe I'm either trying to achieve the impossible or trying it wrong.
TLDR: Is it possible to set up Kafka ACLs to allow using any consumer group without also granting ListGroups API permission at the same time?
Thanks for any answer.
For now decided to use prefix. Works well enough.
For those wondering how to do this:
bin/kafka-acls.sh localhost:9092 --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=zookeeper.address --add --allow-principal User:XYZ --operation Read --group 'ABC-' --resource-pattern-type prefixed
This piece of code will allow user 'XYZ' to use any consumer group starting with 'ABC-', like 'ABC-123'
I am collecting the data from different resources, each resource has one specific topic for each client.
I want to give the access for each user only to the corresponding topic, so they can't have access to all the topics.
I am working with Kafka 0.10 and I am using Kafka tools.
there is solution?
You need to configure Authorisation using ACL.
How to enable ACL:
In your server.properties file, you need to create an Authorizer by adding the following line:
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
Now you need to follow the docs in order to properly configure ACL based on your use cases.
Adding ACLs
Now once everything is in place, let's assume you have a topic called testTopic to which you want to grant read and write access only to user called Bob from a host with IP 197.5.6.1:
bin/kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 \
--add \
--allow-principal User:'Bob' --allow-host '197.5.6.1' \
--operation Read --operation Write \
--topic testTopic \
We have different consumer groups which consumes data from different topics.We have different partitions of the topic.We need to allow some consumer groups to have access to a particular topic so that only those groups can read from the topics.I was trying out Confluent Kafka Access control lists like this:
bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --consumer --topic test-topic --consumer-group Group-1
However as we understand it the ACLs can block access to Users and host addresses.Could we use ACLs to block consumer groups as well? If not is there any command that'll help me do it.
Thanks
ACLs on Consumer Groups isn't at all secure because any consumer can change their own group.id value. ACLs need to work off secure authentication credentials that cannot be spoofed or faked easily like X.509 Digital Certs or SASL/Kerberos credentials