I have a Kafka cluster running with Zookeeper, Confluent Schema registry and Kafka security manager(KSM). KSM, https://github.com/conduktor/kafka-security-manager, is software makes it easy to manager Kafka ACL with a csv file instead of using the command line tool.
The confluent schema registry let us store Avro schema for Kafka. It is currently open and I need to secured it. I want to give every user the READ or GET permission only. I am currently using kubernetes to deploy all the tools.
How can I do that with KSM? Where can I find examples?
Thank you
Kafka ACLs don't apply to the Schema Registry, they would apply to the underlying _schemas topic, which you'd setup in the Registry's configuration
The API itself can be secured using TLS and HTTP Authentication
https://docs.confluent.io/platform/current/schema-registry/security/index.html
give every user the READ or GET permission only.
I don't think you can lock down HTTP method level access to specific users, you'll likely need a proxy for this, but also without POST, there's no way to register topics...
Related
I have created a cluster Amazon MSK. Also, created an EC2 instance and installed Kafka on it to create a topic in Amazon MSK. I am able to produce/consume messages on the topic using Kafka scripts.
I have also installed the IIDR Replication agent on an EC2 instance. The plan is to migrate DB2 table data into the Amazon MSK topic.
In the IDR Management console, I am able to add the IIDR replication server as the target.
Now when creating the subscription, it is asking for ZooKeeper URL and Schema Registry URL. I can get the Zookeeper endpoints from Amazon MSK.
What value to provide for the schema registry URL as there's none created?
Thanks for your help.
If you do not need to specify a schema registry because say you are using a KCOP that generate JSON, just put in a dummy value. Equally if you are specifying a list of Kafka brokers in the kafkaconsumer.propertie and the kafkaproducer.properties files in the CDC instance.conf directory you can put in dummy values for the zookeeper fields.
Hope this helps
Robert
I need to add partitions to an existing Kafka topic.
I'm aware that it is possible to use the /bin/kafka-topics.sh script to achieve this, but I would prefer to do this through the Confluent REST api.
As far as I see there is no documented endpoint in the api reference, but I wonder if someone else here was able to make this work.
Edit: As it does seem to be impossible to use the REST api here, I wonder what the best practice is for adding partitions to an existing topic in a containerized setup. E.g. if there is a custom partioning scheme that maps customer ids to specific partitions. In this case the app container would need to adjust the partition count of the kafka container.
The Confluent REST Proxy has no such endpoint for topic update administration.
You would need to use the shell script or the corresponding AdminClient class that the shell script uses
This is the solution I ended up with:
Create a small http service that is is deployed within the kafka docker image
The http service accepts requests to increase the partition count and directs the requests to the kafka admin scripts (bin/kafka/kafka-topics.sh)
Something similar could have been achieved by using the AdminClient NewPartitions api in the Java Kafka lib. This solution has the advantage that the kafka docker image does not have to be changed, because the AdminClient can connect through network from another container.
For a production setup the AdminClient is preferrable, I decided for the integrated script approach because of the chosen language (rust).
I am using helm to deploy kafka using cp-helm-charts
I have enabled zookeeper, kafka, schema registry and control center components. In control center UI I am able to create a topic and set a schema for the topic. However schema validation is not enabled and it is still possible to write arbitrary text to the topic.
I am trying to enable schema validation as described here
by adding these options to my helm values:
cp-control-center:
configurationOverrides:
"confluent.schema.registry.url": http://data-cp-schema-registry:8081
"confluent.value.schema.validation": true
But it has no effect.
QUESTION:
How to enable schema validation for cp-helm-charts kafka?
The idea is to restrict all the contents that does not match specified schema.
Schema validation is only applicable to Confluent Server (the broker), or the topics, not Control Center container, so you'll need to move that override to the kafka configuration instead (and verify it's using the cp-server image)
It's worth mentioning that that's a paid feature of Confluent Enterprise.
What are the different ways to get Kafka Cluster Audit log to GCP Logging?
Can anyone share more information on how can I achieve it?
Thank you!
Assuming you have access to the necessary topic (from what I understand the Audit topic is not stored on your own cluster), to get data out of Kafka, you need a consumer. This could be in any language.
To get data into Cloud Logging, you need to use its API.
That being said, you could use any compatible pair of Kafka clients & Cloud logging clients that you would be comfortable with.
For example, you could write or find a Kafka Connect Sink connector that wraps the Java Cloud Logging client.
I am using Bluemix message hub service. I need to provide read only access to specific topics to specific users. Question is how can I define ACLs in Bluemix message hub? Apache kafka does provide the means (http://kafka.apache.org/documentation.html#security_authz), but that involves interacting with zookeeper. And I haven't been able to find details to connect to underlying zookeeper behind Bluemix message hub service. Appreciate the responses.
ACLs are currently not supported with MessageHub. As you've correctly noted, we don't give users access to Zookeeper. So at the moment, there is no way to only provide read or write access to a topic.
That said, we isolate each service instance. So if you provision 2 MessageHub instances in different Bluemix spaces, they will be fully isolated and won't be able to see each other topics. That way you could have guaranties that users from instance1 won't be able to read or write from/to topics from instance2. Not what you've asked for but that might help.