I'm trying to create a new kafka topic in confluent cloud, but it gives me an 'authorization failed' error.
give yourself a role CloudClusterAdmin from confluent cloud UI Accounts & access, then you should able to manage cluster.
Related
I have created a cluster Amazon MSK. Also, created an EC2 instance and installed Kafka on it to create a topic in Amazon MSK. I am able to produce/consume messages on the topic using Kafka scripts.
I have also installed the IIDR Replication agent on an EC2 instance. The plan is to migrate DB2 table data into the Amazon MSK topic.
In the IDR Management console, I am able to add the IIDR replication server as the target.
Now when creating the subscription, it is asking for ZooKeeper URL and Schema Registry URL. I can get the Zookeeper endpoints from Amazon MSK.
What value to provide for the schema registry URL as there's none created?
Thanks for your help.
If you do not need to specify a schema registry because say you are using a KCOP that generate JSON, just put in a dummy value. Equally if you are specifying a list of Kafka brokers in the kafkaconsumer.propertie and the kafkaproducer.properties files in the CDC instance.conf directory you can put in dummy values for the zookeeper fields.
Hope this helps
Robert
I am trying to create in Confluent Cloud (Kafka) a MongoDB connector sink with ksqlDB. The problem is that I have the data source and credentials in the AWS Secrets manager.
Is there a way to obtain the secrets with ksqlDB to set the connector properties?
Kafka Connect supports Externalized config for secrets. Whether such an implement exists for AWS, I am not sure, but if not, you'll need to write your own ConfigProvider for it.
Alternatively, there may be alternative solutions like running ksql or just Connect itself in MSK Connect, ECS, EC2, or EKS where you write processes around exposing Secrets Manager data into files or environment variables, which can then be used by Connect's default config providers, then setup ksql externally to point at those Connect instances, or just process the topics it outputs
I'm trying to publish custom db data (derived from Microsoft SQL CDC tables, having a join on other tables -> how it's arrived is for a different day though) to Kafka cluster.
I'm able to publish and consume messages from Apache NiFi -to/from- Apache Kafka.
But I'm unable to do publish messages from Apache NiFi -to- Kafka in Confluent Cloud.
Is it possible to publish/consume messages from Apache NiFi (server-A) to Confluent Cloud using the API Key that's created there?
If yes, what is the corresponding property in Apache NiFi's PublishKafkaRecord's processor and ConsumeKafkaRecord processor?
If no, please share any other idea to overcome the constraint.
Yes, NiFi uses the plain Kafka Clients Java API; it can work with any Kafka environment.
Confluent Cloud gives you all the client properties you will need, such as SASL configs for username + password.
Using PublishKafka_2_6 as an example,
Obviously, "Kafka Brokers" is the Bootstrap Brokers, then you have "Username" and "Password" settings for the SASL connection.
Set "Security Protocol" to SASL_SSL and "SASL Mechanism" to PLAIN.
"Delivery Guarantee" will set producer acks.
For any extra properties, use the + button above the properties for setting "Dynamic Properties" (refer above NiFi docs)
share any other idea to overcome the constraint
Use Debezium (Kafka Connect) instead.
For Kafka cluster hosted in Confluent Cloud, there is an Audit Log cluster that gets created. It seems to be possible to hook a Sink connector to this cluster and drain the events out from "confluent-audit-log-events" topic.
However, I am running into the below error when I run the connector to do the same.
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [connect-offsets]
In my connect-distributed.properties file, I have the settings as :
offset.storage.topic=connect-offsets
offset.storage.replication.factor=3
offset.storage.partitions=3
What extra permission/s needs to be granted so that the connector can create the required topics in the cluster? The key/secret being used in the connect-distributed.properties files is a valid key/secret that is associated to the service account for this cluster.
Also, when I run the consumer in the console using the same key (as above) , I am able to read the audit log events just fine.
It's confirmed that this feature (hooking up a connector to the Audit Log cluster) is not supported at the moment in Confluent Cloud. This feature may be available later this year at some point.
How do I use Kafka Connect adapters with Amazon MSK?
As per the AWS documentation, it supports Kafka connect but not documented about how to setup adapters and use it.
Edit Oct 2021: MSK Connect has been launched, see https://aws.amazon.com/blogs/aws/introducing-amazon-msk-connect-stream-data-to-and-from-your-apache-kafka-clusters-using-managed-connectors/
AFAIK Amazon MSK does not provide managed connectors, so you have to run them yourself. This is done by running the Kafka Connect worker process (a JVM) and then providing it one or more connector configurations to run.
From the point of view of a Kafka Connect worker it just needs a Kafka cluster to connect to; it shouldn't matter whether it's MSK or on-premises, since it's ultimately 'just' a consumer/producer underneath.
You can see more, including a live demo, here: https://rmoff.dev/bbuzz19-kafka-connect
For an example of configuring Kafka Connect to use a cloud-hosted Kafka platform (in this case, Confluent Cloud), see this article.
If you are interested in managed connectors in the Cloud, check out the connectors that are provided in Confluent Cloud.
Disclaimer: I work for Confluent :)
AWS now supports MSK Connect, a new feature of MSK service based on Kafka Connect allowing you to deploy managed Kafka connectors built for Kafka connect
Check the announcement here: https://aws.amazon.com/blogs/aws/introducing-amazon-msk-connect-stream-data-to-and-from-your-apache-kafka-clusters-using-managed-connectors/
There are two aspects to this
Kafka Connect is a framework which should be deployed separately from kafka brokers. MSK only provides kafka brokers. If you want to use Kafka Connect with MSK you would need to use EC2 instances and deploy the kafka binaries.Kafka Connect framework is bundled along with kafka
Coming to connectors if you donot have a confluent subscription or similar - I am afraid your choices get very limited. But having said you can always write your own connectors. Writing new connectors is not that difficult rather you can apply your business specific logic and be on your way quite quickly.