I have a Kafka cluster on which multiple teams create their topics using JAVA client (KafkaAdmin).
Now we want to enable ACLs on those topics. I know using superuser it's possible (using Kafka CLI or Admin Client).
Is it possible to let the Kafka Topic creator be the admin for that topic and the creator should be responsible for their topics ACL?
Related
I'm trying to use Apache Kafka with Clickhouse. I have a kafka username and password which I added to config.xml file in clickhouse files liked this:
<kafka>
<sasl_mechanisms>SCRAM-SHA-256</sasl_mechanisms>
<sasl_username>some_user</sasl_username>
<sasl_password>some_pass</sasl_password>
</kafka>
This way I can use Kafka topics which is available for that one user. How can I use multiple user and use different topics available for different users while using a Kafka Engine in Clickhouse?
Is there a way to configure Kafka user settings while writing Kafka table with SQL in Clickhouse?
https://clickhouse.com/docs/en/engines/table-engines/integrations/kafka/#configuration
each topic can have own settings
<kafka_mytopic>
<sasl_mechanisms>SCRAM-SHA-256</sasl_mechanisms>
<sasl_username>yyyy</sasl_username>
<sasl_password>xxxx</sasl_password>
</kafka_mytopic>
<kafka_mytopic2>
<sasl_mechanisms>SCRAM-SHA-256</sasl_mechanisms>
<sasl_username>ddd</sasl_username>
<sasl_password>zzz</sasl_password>
</kafka_mytopic2>
We are doing a stateful operation. Our cluster is managed. Everytime for internal topic creation , we have to ask admin guys to unlock so that internal topics can be created by the kafka stream app. We have control over target cluster not source cluster.
So, wanted to understand which cluster - source/ target are internal topics created?
AFAIK, There is only one cluster that the kafka-streams app connects to and all topics source/target/internal are created there.
So far, Kafka Stream applications can support connection to only one cluster as defined in the BOOTSTRAP_SERVERS_CONFIG in Stream configurations.
As answered above also, all source topics reside in those brokers and all internal topics(changelog/repartition topics) are created in the same cluster. KStream app will create the target topic in the same cluster as well.
It will be worth looking into the server logs to understand and analyze the actual root cause.
As the other answers suggest there should be only one cluster that the Kafka Stream application connects to. Internal topics are created by the Kafka stream application and will only be used by the application that created it. However, there could be some configuration related to security set on the Broker side which could be preventing the streaming application from creating these topics:
If security is enabled on the Kafka brokers, you must grant the underlying clients admin permissions so that they can create internal topics set. For more information, see Streams Security.
Quoted from here
Another point to keep in mind is that the internal topics are automatically created by the Stream application and there is no explicit configuration for auto creation of internal topics.
As the title suggest I am looking for a way to restrict the Kafka Consumers to publish the data to my kafka topic . For example we can have read only user in the database .
My use case requires me to get the data from some vendors, enrich this data and publish it to our Kafka topic. The data from this topic will be be read by few consumers. And as I understand I have to provide the same username and password ( the one that producer is using ) to the consumers. So is there some way to stop consumers to publish the data in the Kafka topic .
E.g: restricting based on the username/ IP address.
By default, there is no such feature in Kafka.
Apache Ranger and Open Policy Agent are example systems that you can implement much richer ACL policies than the Zookeeper options built-in. In my experience, only Ranger can do IP whitelisting.
I don't think the default Zookeeper ACL policies can do much other than username&password access.
I have Kafka Connect JDBC sink connectors writing to various databases and I'd like to throttle the traffic to one database. The Kafka quotas feature can set a consumer_byte_rate quota for a client ID, but Kafka Connect client IDs look like consumer-1234 and are dynamically assigned to connectors. So if my sink connector is rebalanced, it will be assigned all new client IDs. I tried setting a quota using my sink connector consumer group ID as the client ID, but that doesn't work. Is there any way to set a quota for a Kafka Connect consumer group?
If you upgrade to Apache Kafka 2.3 you'll benefit from KIP-411: Make default Kafka Connect worker task client IDs distinct
. You can see an example of it in action here. However, you'd have to test if the client-id is deterministic since quotas can't be wildcarded.
My setup is the following:
3 Zookeeper nodes secured in the following way:
SASL enabled (quorum.auth.enableSasl=true)
Requires SASL for learners (learnerRequireSasl=true)
Require SASL for servers (quorum.auth.serverRequireSasl=true)
Require SASL for clients (requireClientAuthScheme=sasl)
A jaas.conf file with the entries QuorumServer, QuorumLearner (both with the same zookeeper account and password), and Server (with a kafka plus a superuser account, plus passwords)
The idea of the superuser account is that I can use a separate identities and secrets (and possibly permissions) for the Kafka cluster vs. connections by admins from CLI tools.
Then...
3 Kafka nodes secured in the following way:
All listeners require SASL_PLAINTEXT (listener.security.protocol.map)
SASL mechanism is SCRAM-SHA-512 (sasl.enabled.mechanisms)
Brokers require SASL for interbroker as well as client connections (sasl.mechanism.inter.broker.protocol)
Super users: kafka, superuser
Set ACLs on all metadata that Kafka creates (zookeeper.set.acl=true). See (KIP-38)(https://cwiki.apache.org/confluence/display/KAFKA/KIP-38%3A+ZooKeeper+Authentication)
In Kafka + Zookeeper deployments with default settings, Zookeeper essentially applies no noteworthy protection mechanisms. Any rogue actor who can connect to a Zookeeper instance (e.g. after penetrating the so-called isolated network) can change Kafka metadata stored in Zookeeper at will, such as creating new Kafka users and elevating permissions.
With the zookeeper.set.acl=true setting, Kafka will automatically apply ACLs to all the Znodes it creates (for clusters, topics, offsets, etc.) so that its Znodes are protected from unauthenticated and unauthorized access = more defense in depth.
Important: These ACLs are Znode ACLs (a Zookeeper concept) and not the same as the Kafka ACLs that can be applied to clusters, topics, and the like. The zookeeper-shell.sh example below shows the subnodes of /config and the ACL set on the /config/users Znode. Only the kafka identity has full control, world has no access whatsoever:
ls /config
[changes, clients, brokers, users, topics]
getAcl /config/users
'sasl,'kafka
: cdrwa
Kafka will only set ACLs on Znodes for one account, which is typically named kafka. Certain Kafka administration tasks, such as adding Kafka users with SCRAM-SHA-512 authentication (kafka-configs.sh tool), cannot be done through Kafka brokers but require direct interaction between the CLI tool and Zookeeper.
And this finally gets me to the problem that I am facing: Because Znode ACLs automatically set by Kafka brokers are only set for the kafka identity, it is not possible to perform Zookeeper CLI operations using any other identity, such a superuser identity.
Question: Does anybody know how to make Kafka set Znode ACLs for more than just the kafka identity? Specifically, I would also like the superuser identity to be able to make modifications directly in Zookeeper.